I have a decades long career keeping fleets of services, databases, and infrastructure alive. In these notes, I cover doing the same for real, production LLM inference infrastructure: an overview of what it takes to build, deploy, test, and maintain a serving platform that other teams and organizations can rely on.

Building an LLM inference platform

A series that builds a real, production ready self-hosted inference platform on Kubernetes one step at a time, with the code in a public repo alongside each post. Each part ends at a point you can reproduce.

1. Development environment bootstrapping for an inference platform Pinned tooling, a disposable cluster, and a mock engine that behaves like vLLM at the edges. September 2026.
2. The chart an app team actually uses A values file in, a working endpoint out. Probes, rollouts under load, and tests that run without a cluster.
3. Watching it, then scaling it Prometheus, the metrics that matter for inference, and autoscaling on queue depth instead of CPU.
4. Bringing it to the cloud Argo CD, OpenTofu, and a GPU node that the chart deploys to unchanged.
5. Metrics and real numbers vLLM on rented hardware, benchmarks you can rerun, and the SLOs and thresholds that fall out of them.
6. Running it like a fleet Dozens of mock replicas, replayed production traffic, game days, SLOs with error budgets, and the postmortems that come out of it.

The code is at github.com/samrose/inference-platform. Decisions are recorded in the repo's docs/adr/ directory as they're made.