Development environment bootstrapping for an inference platform

September 2026. Part 1 of Building an LLM inference platform. Code: tag post-1.

This series' goal is not just being a guide or blog post. My aim is to provide a very comprehensive, reproducible approach to building a production-ready LLM inference platform.

The code is at github.com/samrose/inference-platform, and each post has a tag.

What this part covers

Most of an LLM inference infrastructure platform can be built and tested on a laptop without a GPU. The chart, probes, metrics scraping, dashboards, alerts and autoscaling only depend on how the engine behaves at its edges: the network port it listens on, what the health endpoint returns while weights load, and what the metrics are called. This post covers setting up a local tier with a stand-in engine that gets those right.

Pinned tooling

The repo I created for this inference platform needs about a dozen CLIs (kubectl, helm, kind, kubeconform, OpenTofu, argocd, just, python, uv etc). I pin them with a Nix flake for local development, and later for continuous integration. The flake provides reproducible versions for both environments, and the flake.lock is committed. So bumping a version of one of these tools is a reviewable diff (ADR 0001). Nix is only used for tooling here. The production OCI images are built with Docker and charts with Helm. If you don't want Nix, install the versions below some other way. Docker is the one thing the flake doesn't provide (I use OrbStack on a Mac).

$ nix develop
inference-platform dev shell
  kubectl  v1.37.0
  helm     v4.3.0
  kind     v0.32.0
  tofu     v1.12.6
  argocd   refs/tags/v3.4.6

I selected OpenTofu rather than Terraform because of the 2023 license change, since this is a public repo (ADR 0002). Helm 4 and kind 0.32 were chosen because Helm 3 reaches end of life in 2027 and this is a new repo.

A disposable cluster from one command

Every local operation is a recipe in the justfile (just is a command runner if you are not familiar with it). The KUBECONFIG is set to local/kubeconfig in both the justfile and the dev shell, so nothing here can touch a cluster in ~/.kube/config.

just up creates a 3 node kind cluster (one control plane and two workers, so that scheduling across nodes and drains can be tested later), with the node image pinned by digest from the kind v0.32.0 release notes. This command also starts a registry:2 container on 127.0.0.1:5001, and connects it to the kind Docker network + configures containerd on each node to pull localhost:5001 images from it. I decided to run a local registry over kind load docker-image so that the local workflow is build, push, reference by digest. The benefit is that the workflow will be the same as it will be in the cloud (ADR 0003).

just down removes the cluster and keeps the registry, just nuke removes both. just up takes about a minute and is safe to run repeatedly:

$ just up
cluster 'inference' already exists
just _connect-registry
configmap/local-registry-hosting unchanged
kubectl wait --for=condition=Ready nodes --all --timeout=120s
node/inference-control-plane condition met
node/inference-worker condition met
node/inference-worker2 condition met
just status
kubectl get nodes -o wide
NAME                      STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION                                CONTAINER-RUNTIME
inference-control-plane   Ready    control-plane   28h   v1.36.1   192.168.148.2   <none>        Debian GNU/Linux 13 (trixie)   7.0.14-orbstack-00380-ga7e0a2dc9535 (arm64)   containerd://2.3.1
inference-worker          Ready    <none>          28h   v1.36.1   192.168.148.4   <none>        Debian GNU/Linux 13 (trixie)   7.0.14-orbstack-00380-ga7e0a2dc9535 (arm64)   containerd://2.3.1
inference-worker2         Ready    <none>          28h   v1.36.1   192.168.148.3   <none>        Debian GNU/Linux 13 (trixie)   7.0.14-orbstack-00380-ga7e0a2dc9535 (arm64)   containerd://2.3.1
registry: localhost:5001 (running)

# justfile — one entrypoint for every local operation.
# Run `just` to list recipes.

set shell := ["bash", "-euo", "pipefail", "-c"]
set dotenv-load := true

# ---- configuration (override with `just VAR=value recipe` or a .env file) ----
cluster      := env_var_or_default("CLUSTER_NAME", "inference")
reg_name     := env_var_or_default("REGISTRY_NAME", "kind-registry")
reg_port     := env_var_or_default("REGISTRY_PORT", "5001")
kind_config  := "local/kind-config.yml"
kubeconfig   := "local/kubeconfig"
chart        := "charts/inference-service"

export KUBECONFIG := kubeconfig

# default: show available recipes
default:
    @just --list --unsorted

# ---- lifecycle ----------------------------------------------------------------

# create the local registry, the kind cluster, and wire them together
up: registry
    @if kind get clusters | grep -qx "{{cluster}}"; then \
        echo "cluster '{{cluster}}' already exists"; \
    else \
        kind create cluster --name "{{cluster}}" --config "{{kind_config}}" --kubeconfig "{{kubeconfig}}"; \
    fi
    just _connect-registry
    kubectl wait --for=condition=Ready nodes --all --timeout=120s
    just status

# delete the cluster (registry is left running; `just nuke` removes both)
down:
    kind delete cluster --name "{{cluster}}"
    rm -f "{{kubeconfig}}"

# delete the cluster and the registry container
nuke: down
    docker rm -f "{{reg_name}}" >/dev/null 2>&1 || true

# start the local OCI registry if it isn't running
registry:
    @if [ "$(docker inspect -f '{{{{.State.Running}}' "{{reg_name}}" 2>/dev/null)" != "true" ]; then \
        docker run -d --restart=always -p "127.0.0.1:{{reg_port}}:5000" \
            --network bridge --name "{{reg_name}}" registry:2; \
    fi

# tell containerd on every node about the registry and advertise it to tooling
_connect-registry:
    #!/usr/bin/env bash
    set -euo pipefail
    reg_dir="/etc/containerd/certs.d/localhost:{{reg_port}}"
    for node in $(kind get nodes --name "{{cluster}}"); do
        docker exec "$node" mkdir -p "$reg_dir"
        cat <<EOF | docker exec -i "$node" cp /dev/stdin "$reg_dir/hosts.toml"
    [host."http://{{reg_name}}:5000"]
    EOF
    done
    if [ "$(docker inspect -f='{{{{json .NetworkSettings.Networks.kind}}' "{{reg_name}}")" = "null" ]; then
        docker network connect kind "{{reg_name}}"
    fi
    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: local-registry-hosting
      namespace: kube-public
    data:
      localRegistryHosting.v1: |
        host: "localhost:{{reg_port}}"
        help: "https://kind.sigs.k8s.io/docs/user/local-registry/"
    EOF

# cluster and registry health at a glance
status:
    kubectl get nodes -o wide
    @echo "registry: localhost:{{reg_port}} ($(docker inspect -f '{{{{.State.Status}}' "{{reg_name}}" 2>/dev/null || echo absent))"

# ---- verification ---------------------------------------------------------------

# prove the registry round-trips: push an image, run it in the cluster
# (no --platform: Docker pulls the host arch, which is what the kind nodes run)
smoke:
    #!/usr/bin/env bash
    set -euo pipefail
    img="localhost:{{reg_port}}/busybox:1.36"
    docker pull busybox:1.36
    docker tag busybox:1.36 "$img"
    docker push "$img"
    kubectl delete pod smoke --ignore-not-found >/dev/null
    kubectl run smoke --restart=Never --image="$img" -- echo "registry round-trip OK"
    if ! kubectl wait pod/smoke --for=jsonpath='{.status.phase}'=Succeeded --timeout=60s; then
        echo "--- smoke pod did not succeed; describe follows ---"
        kubectl describe pod smoke | sed -n '/Events:/,$p'
        kubectl delete pod smoke --ignore-not-found >/dev/null
        exit 1
    fi
    kubectl logs smoke
    kubectl delete pod smoke >/dev/null
# ---- chart quality gates (filled in as step 1 progresses) -----------------------

# static checks: lint, render, validate against the cluster's API schemas
lint:
    helm lint "{{chart}}"
    helm template test "{{chart}}" | kubeconform -strict -summary -kubernetes-version "$(just _k8s-version)"

# unit tests for the chart's templates
test:
    helm unittest "{{chart}}"

# everything CI runs
check: lint test

# ---- helpers --------------------------------------------------------------------

_k8s-version:
    @kubectl version -o json | jq -r '.serverVersion.gitVersion' | sed 's/^v//'

# ---- mock engine ------------------------------------------------------------------

mock_image := "localhost:" + reg_port + "/mock-engine"

# build the mock engine for the local arch and push it; records the digest
mock-push:
    #!/usr/bin/env bash
    set -euo pipefail
    tag="{{mock_image}}:dev"
    docker build -t "$tag" mock-engine
    docker push "$tag"
    digest=$(docker inspect --format '{{{{index .RepoDigests 0}}' "$tag" | cut -d@ -f2)
    echo "$digest" > local/mock-engine.digest
    echo "pushed {{mock_image}}@$digest"

# run the mock engine in the cluster by digest and exercise every endpoint
mock-smoke:
    #!/usr/bin/env bash
    set -euo pipefail
    img="{{mock_image}}@$(cat local/mock-engine.digest)"
    kubectl delete pod mock --ignore-not-found >/dev/null
    kubectl run mock --restart=Never --image="$img" --port=8000 \
        --env STARTUP_DELAY_SECONDS=5 --env TTFT_MS=100 --env TPOT_MS=10
    kubectl wait pod/mock --for=condition=Ready --timeout=60s
    kubectl port-forward pod/mock 18000:8000 >/dev/null 2>&1 & pf=$!
    trap 'kill $pf; kubectl delete pod mock >/dev/null' EXIT
    sleep 2
    echo "--- health (expect 503 until startup delay elapses, then 200)"
    for i in 1 2 3 4 5 6; do curl -s -o /dev/null -w "%{http_code}\n" localhost:18000/health; sleep 1; done
    echo "--- streamed completion"
    curl -sN localhost:18000/v1/chat/completions -H 'content-type: application/json' \
        -d '{"model":"mock/mock-8b","stream":true,"max_tokens":5,"messages":[{"role":"user","content":"hi"}]}'
    echo "--- admin: force queue depth"
    curl -s -X POST localhost:18000/admin/state -H 'content-type: application/json' -d '{"waiting": 12, "kv_cache_usage": 0.85}'
    echo; echo "--- metrics (vllm:* only)"
    curl -s localhost:18000/metrics | grep '^vllm:' | grep -v '_bucket'

# local/kind-config.yml — the local tier's cluster definition.
# Pinning: the node image is pinned by digest; change it deliberately via an ADR.
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: inference

# Tell containerd on every node to read per-registry config from certs.d.
# `just _connect-registry` writes /etc/containerd/certs.d/localhost:5001/hosts.toml
# into each node so pulls from localhost:5001 resolve to the kind-registry container.
containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"

nodes:
  - role: control-plane
    # Take the full image string for your chosen Kubernetes minor from the
    # kind v0.32.0 release notes ("Images built for this release").
    image: kindest/node:v1.36.1@sha256:3489c7674813ba5d8b1a9977baea8a6e553784dab7b84759d1014dbd78f7ebd5
    # Label the control plane so a gateway/ingress can be pinned to it later.
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    # Host ports for the gateway (step 5). Unused until then; harmless now.
    extraPortMappings:
      - containerPort: 80
        hostPort: 8080
        protocol: TCP
      - containerPort: 443
        hostPort: 8443
        protocol: TCP

  - role: worker
    image: kindest/node:v1.36.1@sha256:3489c7674813ba5d8b1a9977baea8a6e553784dab7b84759d1014dbd78f7ebd5

  - role: worker
    image: kindest/node:v1.36.1@sha256:3489c7674813ba5d8b1a9977baea8a6e553784dab7b84759d1014dbd78f7ebd5

The first bug encountered

The first bug I encountered writing this dev env was related to how just handles its own variables. just uses {{ }} for its own variables, and so does docker inspect --format. To pass a literal {{ through just you write {{{{, and the closing }} is left alone. I escaped both sides:

if [ "$(docker inspect -f='{{{{json .NetworkSettings.Networks.kind}}}}' "{{reg_name}}")" = "null" ]; then
    docker network connect kind "{{reg_name}}"
fi

Docker receives {{json .NetworkSettings.Networks.kind}}}} and treats the trailing }} as text, so for a container that is not on the kind network it prints null}} instead of null. The comparison fails and docker network connect never runs. No error, exit code 0, and just up looks healthy.

just smoke then timed out. The recipe used kubectl run --rm -i, which reports a bare timeout when a pod never starts and deletes the pod afterwards, so there was nothing to inspect. A wrong image, wrong architecture, or unreachable registry would all have looked identical.

The pod events had the answer. This is what they look like with the registry disconnected from the kind network:

error: timed out waiting for the condition on pods/smoke-repro
--- smoke pod did not succeed; describe follows ---
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  45s                default-scheduler  Successfully assigned default/smoke-repro to inference-worker
  Normal   BackOff    21s (x2 over 44s)  kubelet            Back-off pulling image "localhost:5001/busybox:repro-1789847397"
  Warning  Failed     21s (x2 over 44s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    10s (x3 over 45s)  kubelet            Pulling image "localhost:5001/busybox:repro-1789847397"
  Warning  Failed     10s (x3 over 45s)  kubelet            Failed to pull image "localhost:5001/busybox:repro-1789847397": failed to pull and unpack image "localhost:5001/busybox:repro-1789847397": failed to resolve reference "localhost:5001/busybox:repro-1789847397": failed to do request: Head "http://kind-registry:5000/v2/busybox/manifests/repro-1789847397?ns=%5BREDACTED%5D": dial tcp: lookup kind-registry on 0.250.250.254:53: no such host
  Warning  Failed     10s (x3 over 45s)  kubelet            Error: ErrImagePull

The long line ends with:

failed to do request: Head "http://kind-registry:5000/v2/busybox/manifests/repro-1789847397?ns=%5BREDACTED%5D": dial tcp: lookup kind-registry on 0.250.250.254:53: no such host

The pod events showed the pull failing because the node couldn't resolve kind-registry by name.

The fix was removing two characters. I also changed smoke so it waits 60 seconds for the pod and prints the events if it fails:

if ! kubectl wait pod/smoke --for=jsonpath='{.status.phase}'=Succeeded --timeout=60s; then
    echo "--- smoke pod did not succeed; describe follows ---"
    kubectl describe pod smoke | sed -n '/Events:/,$p'
    kubectl delete pod smoke --ignore-not-found >/dev/null
    exit 1
fi

A timeout told me something was wrong but not what. We really need failing checks in this repo to say why, and the same consideration will apply to probes and alerts later on in this series and platform I create.

A mock engine

To build everything above the engine without a GPU, I need something that behaves like vLLM at the points where the platform touches it:

• OpenAI-compatible /v1/chat/completions with SSE streaming, and /v1/models, on port 8000 with the same paths as vllm serve, so the chart needs no mock-specific branches
• /health returns 503 until the weights are loaded, then 200. The mock takes a STARTUP_DELAY_SECONDS
• /metrics with vLLM's metric names, labels and histogram buckets, so PromQL written against the mock works unchanged against the real engine

mock-engine/ is a FastAPI app of about 190 lines that does this (ADR 0004).

I considered vLLM on CPU with a small model. The mock starts in under a second and answers in milliseconds, so the feedback loop is instant. It emits a deliberately chosen subset of vLLM's metrics, with the same names, labels, and buckets, so every query written against it works unchanged in production. To test autoscaling on queue depth I need to set the queue to 12 and see what happens. So the mock has POST /admin/state, which sets queue depth and KV cache usage. Unlike queue depth and cache usage, which the admin endpoint fakes, the num_requests_running gauge is genuinely counted: it goes up when a request arrives and down when the response finishes. So concurrent requests to the mock show up in that metric the same way they would on a real engine.

One detail worth noting: prometheus_client adds a _created series for every counter and histogram by default, and vLLM's output doesn't have them. The Dockerfile sets PROMETHEUS_DISABLE_CREATED_SERIES=true so that the mock's metrics stay a subset of the real engine's, otherwise a dashboard could end up depending on a series that isn't there in production.

just mock-push builds and pushes the image and records its digest. just mock-smoke runs it in the cluster with a 5 second startup delay and hits each endpoint:

$ just mock-smoke
pod/mock created
pod/mock condition met
--- health (expect 503 until startup delay elapses, then 200)
503
503
503
503
200
200
--- streamed completion
data: {"id": "chatcmpl-50193b636528", "object": "chat.completion.chunk", "created": 1789847337, "model": "mock/mock-8b", "choices": [{"index": 0, "delta": {"role": "assistant", "content": ""}, "finish_reason": null}]}

data: {"id": "chatcmpl-50193b636528", "object": "chat.completion.chunk", "created": 1789847337, "model": "mock/mock-8b", "choices": [{"index": 0, "delta": {"content": "tok0 "}, "finish_reason": null}]}

[tok1 through tok4 omitted]

data: {"id": "chatcmpl-50193b636528", "object": "chat.completion.chunk", "created": 1789847337, "model": "mock/mock-8b", "choices": [{"index": 0, "delta": {}, "finish_reason": "length"}]}

data: [DONE]

--- admin: force queue depth
{"ready":true,"waiting":12,"kv_cache_usage":0.85}
--- metrics (vllm:* only)
vllm:num_requests_running{engine="0",model_name="mock/mock-8b"} 0.0
vllm:num_requests_waiting{engine="0",model_name="mock/mock-8b"} 12.0
vllm:kv_cache_usage_perc{engine="0",model_name="mock/mock-8b"} 0.85
vllm:gpu_cache_usage_perc{engine="0",model_name="mock/mock-8b"} 0.85
vllm:time_to_first_token_seconds_count{engine="0",model_name="mock/mock-8b"} 1.0
vllm:time_to_first_token_seconds_sum{engine="0",model_name="mock/mock-8b"} 0.10264143301174045
vllm:time_per_output_token_seconds_count{engine="0",model_name="mock/mock-8b"} 5.0
vllm:time_per_output_token_seconds_sum{engine="0",model_name="mock/mock-8b"} 0.05
vllm:e2e_request_latency_seconds_count{engine="0",model_name="mock/mock-8b"} 1.0
vllm:e2e_request_latency_seconds_sum{engine="0",model_name="mock/mock-8b"} 0.15268072301114444
vllm:prompt_tokens_total{engine="0",model_name="mock/mock-8b"} 1.0
vllm:generation_tokens_total{engine="0",model_name="mock/mock-8b"} 5.0
vllm:request_success_total{engine="0",finished_reason="length",model_name="mock/mock-8b"} 1.0

Note pod/mock condition met followed by four 503s. The pod is Ready before the engine is, because a pod from kubectl run has no probes. With a real engine that gap is as long as the weights take to load, and the pod would be receiving traffic during it. The chart in part 2 fixes that.

What this mock engine can't tell you

The mock set up doesn't batch, it doesn't model latency under load, and it doesn't tokenize. The time to first token and time per output token aren't measured; they're whatever numbers I set in environment variables, and the mock just sleeps for that duration. No performance conclusions can come from it. Benchmarks come later, from the real engine on real hardware.

The local tier is arm64 on my machine and cloud GPU nodes are amd64. just mock-push builds for the host only, so the digest in local/mock-engine.digest won't work in the cloud. Multi-arch builds come with the cloud tier (ADR 0005).

Next

In Part 2 I will provide the Helm chart to hold the values file an app team will use to put a model into service onto Kubernetes. In addition I'll include startup probe timing, chart tests that run without a cluster, and a rolling update under load.