This one’s shorter than usual — less narrative, more a list I wish someone had handed me before I started running inference on Moya. If you’re about to do the same thing on your own cluster, save yourself the weekends I lost.
1. Your GPU desktop is now a pet, not cattle
Kubernetes culture tells you to treat nodes as disposable cattle, not pets. That’s mostly right, except your GPU-equipped desktop is functionally a pet whether you like it or not — there’s only one (or two, in my case) and replacing it isn’t a kubectl drain away, it’s an “order new hardware and wait” away. Plan your taints, labels, and monitoring accordingly. Treat it as disposable in your manifests, but treat it as precious in your monitoring.
2. Model files are not container images
I covered this in the last post but it bears repeating as its own rule: don’t bake weights into images. Use a persistent volume. Your future self, re-pulling a 9GB image because a pod restarted, will not thank present-you.
3. Resource limits are not suggestions
Setting limits “roughly right” and assuming Kubernetes will gracefully degrade is how you get a silent OOM-kill restart loop. Watch actual memory usage under real load before locking in limits, and set up restart-count alerting from day one — not after you’ve lost a weekend to a quietly crash-looping pod, like I did.
4. Quantization is a real engineering decision, not an afterthought
It’s tempting to grab whatever quantized model someone else uploaded and move on. It’s worth actually testing a couple of quantization levels against your real workloads first. The gap between “fine for casual chat” and “fine for RAG over my own documentation” is bigger than I expected going in. Running multiple instances has also afforded me the ability to run several smaller models that fit entirely on the GPUs. I can have an OCR only model running entirely in VRAM working while the chat model is on an entirely different GPU, and both can task at the same time.
5. GitOps pays off disproportionately for AI workloads specifically
Of everything I run on Moya, the AI inference stack is the thing I’ve been most grateful to have fully declared in Git. Model serving configs, PVC definitions, node affinity rules — there are a lot of moving, interdependent pieces, and hand-editing any of them live is asking for drift between what you think is running and what’s actually running. ArgoCD catching that drift automatically has saved me more than once.
6. Start smaller than your hardware allows
I was tempted to go straight for the largest model my VRAM could technically hold. Started smaller instead, got the Kubernetes scheduling and storage patterns solid first, then scaled the model size up once the plumbing wasn’t in question. Debugging “is this a model problem or an infrastructure problem” at the same time is no fun, and bare-metal K8s gives you plenty of infrastructure problems to debug on their own.