Best laid plans

Jun 7, 2024 m73 lenovo homelab kubernetes hardware mpcie ethernet

Putting together a homelab Kubernetes cluster in my own stubborn way. I’m assuming a reader who’s basically me before I embarked on this little expedition, so I won’t go into minute detail about day-to-day Linux setup and administration - only the things that are new to me and have changed since I last encountered them.

• Part 0 - Best laid plans
• Part 1 - Installing the hardware
• Part 2 - Boot across the network
• Part 3 - PXE Booting Debian with an NFS Root Filesystem
• Part 4 - Filesystems for everybody!
• Part 5 - Kubernetes at last

Sections added as I proceed with this!

The genesis

I’m mostly an AWS flavoured person at work, but most of our stuff runs under Kubernetes (a mixture of self-managed and EKS now) and I don’t know that as well as I would like to. Of course I’ve played with minikube, and I did set up a tiny cluster in the cloud sandbox on Linux Academy¹ following their CKAD course back before they got bought by ACloudGuru and they in turn by PluralSight.

Still, I had a hankering to run on something a bit more like a real cluster. Plus I want to try things like yanking the network cable out of a worker instance and seeing how well it handles it. People prefer it if you don’t do that sort of thing at work.

Anyway, with these vague thoughts in mind I saw this listing on eBay:

At £15 per unit … and not a lot more with postage and packaging… that seems like too good a bargain to miss. They’re only populated with 4G of RAM, but I happened to have a small pile of suitable 8G SODIMMs from an earlier project. No disks, but wouldn’t it be fun to have them netboot and use my Synology NAS for their storage. Could I build a little cluster out of these? I mean, it would be rude not to…

They turned up, well packed with no damage, and they’re pretty solidly constructed little machines anyway. What’s inside the tin though?

Specifications

These are not mighty powerhouses.

• Mine have an Intel i3-4150T CPU running at 3GHz with support for 2 cores and 4 threads.
• The onboard Ethernet controller supports 1Gbit.
• As supplied there’s a measly 4G RAM and no storage. Two SODIMM slots in each machine.
• There’s no WiFi hardware (either removed or never installed)

The basic upgrades

I had enough 8G RAM SODIMMS kicking around that I could bump eight of them up to 16G RAM and of course that left me with plenty of 4G SODIMMS so the remaining two got 8G apiece.

One of the machines would end up being the “cluster gateway” and would need a disk and two network ports. The disk is easy, I had a suitable 512G SATA SSD of indeterminate speed, that went in no problem.

That second network port… well, there was a blanking plate on the back for something; probably a serial port or something? Surely I could mount an ethernet port there if I had something to connect it to inside?

Inside there’s an unpopulated Mini PCI Express (mpcie) slot that’s normally occupied by the wifi card. Using this to drive a wired ethernet port appealed to me as it would be capable of supporting fast speeds and I wouldn’t have an annoying USB dongle hanging off the back of the machine. I bought a suitable mpcie Ethernet board on Swedish Amazon and crossed my fingers that it would be ok in my system.

It is possible to upgrade the CPU in these machines (even as far as a Xeon apparently) but for now I’m not looking for a performance beast, and the upgrade cost would more than double the “investment” in them so far - £30 a pop! Perhaps I’ll upgrade that later if I’m still having fun with it.

Uh, yeah, the plan

So, here’s the plan…

The Telia Router is my home internet router. We get symmetric gigabit internet out of that thing (yay, Sweden!) which is damned useful as both my wife and I work from home a lot of the time. Anyway that’s our gateway to the internet and it needs to be accessible from the machines in the cluster I’m setting up.

That router’s configurable, but out of the box it provides dhcp for a subnet of 192.168.1.0/24 and itself sits and listens internally to the subnet on 192.168.1.1 as the gateway. It NATs the internal subnet to the real internet and it does get assigned a public IP address and one can route on a per-port basis to things inside the subnet (not that I need that much). I won’t be changing any of that. What it doesn’t supply is any fancy dhcp flags. This includes options 66 and 67 which are what you need for netboot. So if I want to netboot my cluster, I’m going to need to set up something custom.

That’s where the PC I’m upgrading to have two ethernet ports comes in. This is my “cluster gateway server” and it’s going to look like this:

On the built-in Ethernet port it’s going to look like a fairly normal client PC on the Telia router’s subnet.

On the newly added Ethernet port it’s going to host a bunch of services:

• A DHCP server
  • To hand out IP addresses to the Kubernetes worker nodes
  • To provide the necessary information for PXE² network booting
• A TFTP server
  • The source of bootstrap files for network boot
• An NFS server
  • This is where the worker node will have its root filesystem as that won’t have its own storage
  • Later I’ll replace this by accessing my Synology NAS indirectly
• Anything and everything that’s needed for Kubernetes but can be independent of the worker nodes
  • I’m not looking to build a particularly resilient system here; a single point of failure is fine

As well as those services - which are all primarily visible on the new Kubernetes subnet, the gateway will do two other important things:

• Packet forwarding
  • So that data can get from one network to the other (if it has the right destination IP address) the router will forward packets between the networks
• Network Address Translation (NAT)
  • The Telia Router knows nothing about my Kubernetes subnet; by using the gateway as a NAT gateway, it will look like all the traffic comes from that one machine and inbound packets on connections made from the Kubernetes network can find their way home.

Once that’s all set up I’ll do one other tweak; I will need to add the cluster gateway as an entry in my laptop’s route table. Otherwise my laptop will send them to its default route, which means it will pass them to the Telia Router… which knows nothing about that subnet and will reject them.

Basically I will tell my laptop:

“If you want to send traffic to 192.168.0.someting then give it to 192.168.1.157 and that machine knows what to do with it.”

I think this will all work, but there are some big fuzzy patches around the following:

• How do I get a worker node to bootstrap so that it’s only using NFS for its root filesystem?
• What exactly do I need installed on the cluster gateway to make the Kubernetes stuff work right? Just the API server? Or other stuff?

I’m sure there will be other surprises along the way.

Next

Next up, collating all this hardware and making sure it works.

Part 1 - Installing the hardware

Footnotes

¹ I enjoyed that LinuxAcademy course, but I didn't bother to sit for the certification because the proctoring process (install shitty software on your laptop and clear your room of all material) sounds so hideous. At least I can (and do) take the AWS exams at one of their partner's premises using their hardware instead of mine.

² "Pre-Boot eXecution Environment", which used to be called "NetBoot" ... basically it's a bunch of BIOS extensions that knows how to do basic DHCP configuration, how to use that to find a TFTP server on the subnet, and how to use that to find, load, and run a binary.