Last post I covered the why behind running private AI on Moya. This one is the how — specifically, getting GPU-backed inference working cleanly on a bare-metal Kubernetes cluster instead of just running a model on a desktop with a script and calling it a day.
If you’ve only ever run AI workloads on cloud-managed Kubernetes (EKS, GKE, etc.), bare metal removes a lot of the conveniences you didn’t know you were relying on. Here’s what I had to build myself.
Getting the GPU Visible to Kubernetes at All
By default, a pod scheduled on a node with a GPU doesn’t know the GPU exists. You need the NVIDIA device plugin (or the equivalent for your hardware) running as a DaemonSet so the kubelet can advertise the GPU as a schedulable resource. Once that’s in place, a pod can request it like any other resource:
resources:
limits:
nvidia.com/gpu: 1The gotcha: this only runs on nodes where the plugin is actually deployed and the drivers are installed at the OS level first. I learned this the hard way when a pod sat in Pending for thirty minutes because I’d forgotten to label the desktop nodes and the scheduler had nowhere valid to put it.
Forcing Workloads Onto the Right Nodes
Moya’s three mini PCs have no business running a 13B parameter model — they’d choke instantly. I use node affinity plus taints/tolerations to make sure inference pods only ever land on the two desktop nodes:
- Taint the GPU nodes so nothing schedules there by default
- Add a matching toleration only to the inference deployment
- Pair it with
nodeAffinityagainst a label likehardware: gpu-desktopfor belt-and-suspenders certainty
This combo matters more than it sounds like it should. Taints alone stop accidental scheduling; affinity alone doesn’t stop other workloads from sneaking onto your GPU box and eating RAM you need for model weights. You want both.
Storage: Don’t Let Model Files Live in the Pod
Quantized model weights are large enough that baking them into a container image is a bad time — slow pulls, bloated registries, and you’re re-downloading multi-gigabyte files every time a pod restarts. I mount a persistent volume backed by Pilot’s TrueNAS storage instead, so the model files live once on the NAS and any inference pod that gets scheduled can attach to them.
This is also where the multi-cluster architecture actually pays off: if I want to test a new model or quantization on Talyn before promoting it to Moya, the same PVC pattern applies, and ArgoCD keeps the manifests in sync so I’m not hand-copying YAML between clusters.
Quantization: Where I Landed
I won’t pretend to be exhaustive here, but the short version of my experience: full-precision weights are not realistic on consumer-grade GPUs sitting in desktop towers, and going too aggressive on quantization (low-bit) introduces noticeable quality loss for anything beyond simple tasks. I settled on a middle ground that keeps inference fast and memory-reasonable without the model feeling noticeably dumber than the unquantized version for my actual use cases — RAG queries against my own notes, rewrites, and quick classification tasks. Your mileage will vary depending on your GPU’s VRAM ceiling, so test on your own hardware rather than trusting a blog post’s specific numbers (including this one).
The Weekend I Lost to Resource Limits
Worth admitting: I once set memory limits on the inference pod too conservatively, assuming Kubernetes would gracefully throttle. It doesn’t — it OOM-kills the pod, which then restarts, which then tries to reload multi-gigabyte model weights from the PVC, which takes a few minutes, which then gets killed again because the limit is still too low. That loop ran quietly for most of a weekend before I noticed inference had effectively been down the whole time. Lesson: set resource requests and limits based on actual observed usage under load, not a guess, and set up basic alerting on pod restart counts. I hadn’t bothered with that last part yet. I have now.