Verification note: every command, file path, version and number below comes from the llm-d v1.0.0 release notes, the optimized-baseline guide on the release-1.0 branch, the branch’s guides/env.sh and the llm-d v1.0 announcement. We did not run the 4-GPU reference deployment on 7minai hardware, so this guide makes no throughput or latency claims of its own.

llm-d 1.0 shipped on October 20, 2026, and it is now the simplest supported way to put an LLM-aware load balancer in front of a pool of vLLM (or SGLang) servers on Kubernetes. The fastest route is the project’s “optimized baseline”: a Helm-installed llm-d Router plus a Kustomize overlay that runs two vLLM replicas of Qwen/Qwen3-32B. The router sends requests that share a prompt prefix to the server that already has that prefix cached, and spreads load by token count rather than round-robin. You need a cluster with 4 GPUs for the reference setup (2 replicas with tensor parallelism 2 on H100 80 GB), a Hugging Face token, and about six commands. This guide walks through them, shows how to prove the prefix-cache routing is actually working, and lists the 0.x-to-1.0 changes that will break an existing setup.

Key facts (from the release notes and announcement):

• Release: llm-d v1.0.0, published October 20, 2026. All core components are now aligned at v1.0.0.
• What’s new: priority- and fairness-aware flow control under one flowControl config, router high availability with a hot standby, DisaggregatedSet for prefill/decode rollouts, the router configuration API graduating to v1, and previous_response_id support in the Responses API.
• Default engine images: vLLM v0.30.0 (docker.io/vllm/vllm-openai), SGLang v0.5.22.
• Pinned infrastructure: Gateway API Inference Extension (GAIE) CRDs v1.6.2, router Helm chart v1.0.0.
• Deprecated: the ghcr.io/llm-d/llm-d-rocm image. llm-d is moving to upstream engine images.

If you have been scaling a single llama-server instead, our guide on serving many users with llama-server covers that end of the spectrum. llm-d is for the next step up: several model-server replicas behind one endpoint.

---

1. What llm-d actually does (and when you need it)

A normal Kubernetes Service balances requests round-robin. That works for short, uniform HTTP calls, but the optimized-baseline guide points out that LLM requests are multi-turn (the same growing prefix arrives again and again), slow (one request can generate for over a minute) and non-uniform (from thousands of reasoning tokens to 100k+ context tokens). Round-robin sends the second turn of a conversation to a server that never saw the first one, so the prefix has to be recomputed.

The llm-d Router sits in front of your model servers. Its Endpoint Picker (EPP) chooses a pod for each request. In the optimized baseline it uses two plugins with their built-in defaults:

• prefix-cache-affinity-filter narrows the candidates to “sticky” pods that probably already cache the prompt prefix. A saturation-aware override spreads load again when those pods get hot.
• token-load-scorer scores pods by the total prefill token load each one is handling.

The project’s announcement cites production results from adopters: Google Cloud reports more than 35% lower time-to-first-token in Vertex AI and a prefix-cache hit rate that doubled from 35% to 70% using the llm-d Endpoint Picker, and IBM Research reports about a 98% prefix-cache hit rate serving up to 3,000 concurrent coding agents. Those are their numbers on their workloads, not a benchmark you will reproduce with this smoke test.

You probably need llm-d if you run two or more replicas of the same model and your traffic is chat or agent-style with long shared prefixes. You probably don’t if you serve one replica on one machine. In that case a single vLLM or llama.cpp server, like the setup in our Qwen3.8-Flash-Next vLLM guide, is simpler.

2. Prerequisites

Cluster and accelerators

The reference configuration is NVIDIA GPUs serving Qwen/Qwen3-32B with 2 replicas × tensor parallelism 2 = 4 GPUs, measured on H100 80 GB. The guide also ships validated overlays for AMD Instinct MI355X (same model, 4 GPUs), Intel XPU (Qwen/Qwen3-0.6B) and Google TPU v5p, plus community overlays for CPU, TPU v6e/v7 and several other accelerators. Each accelerator serves exactly one model in this guide:

ACCELERATOR_TYPE	Served model	Status
gpu (NVIDIA)	Qwen/Qwen3-32B	vLLM and SGLang validated (nightly E2E)
amd	Qwen/Qwen3-32B	vLLM, SGLang and Atom validated
xpu (Intel)	Qwen/Qwen3-0.6B	vLLM validated
tpu/v5p	Qwen/Qwen3-32B	vLLM validated, GKE only
cpu	meta-llama/Llama-3.2-3B-Instruct	community; needs AMX or AVX512-BF16

The model must match your accelerator. If you set ACCELERATOR_TYPE=cpu but leave MODEL=Qwen/Qwen3-32B, your test requests will name a model the pods are not serving.

DRA-based overlays (Intel XPU, Rebellions NPU) use the resource.k8s.io/v1 APIs and need Kubernetes 1.34+.

Client tools

The client-setup page lists the minimum versions. The repo also includes an install-deps.sh script that installs them:

Tool	Minimum version
kubectl	v1.28.0
helm	v3.12.0
kustomize	v5.0.0
git	v2.30.0
yq	v4
helmfile	v1.1.0

A Hugging Face token

The model servers download weights with a token stored in a Secret named llm-d-hf-token. Create one at Hugging Face and export it:

export HF_TOKEN=<your Hugging Face token>

3. Clone the release branch and set the environment

All manifests, Helm values and overlays live next to the guide, so you work from a clone of the llm-d repository on the release-1.0 branch:

export BRANCH=release-1.0
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${BRANCH}

Set the guide variables. These are the official defaults for NVIDIA GPUs and vLLM:

export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
export GUIDE_NAME=optimized-baseline
export NAMESPACE=llm-d-optimized-baseline
export MONITORING=false
export MONITORING_VALUES=
export ACCELERATOR_TYPE=gpu      # gpu, amd, xpu, tpu/v5p, tpu/v6, tpu/v7, cpu, ...
export MODEL_SERVER=vllm         # vllm, sglang, trtllm, atom
export INFRA_PROVIDER=base       # base, gke, amd-ci
export MODEL=Qwen/Qwen3-32B
source ${REPO_ROOT}/guides/env.sh

env.sh is what pins the release. On release-1.0 it sets GAIE_VERSION=v1.6.2, ROUTER_CHART_VERSION=v1.0.0, ROUTER_STANDALONE_CHART=oci://ghcr.io/llm-d/charts/llm-d-router-standalone, ROUTER_GATEWAY_CHART=oci://ghcr.io/llm-d/charts/llm-d-router-gateway and CURL_TEST_IMAGE=cfmanteiga/alpine-bash-curl-jq:latest.

It also has a guard. If the guide has no overlay for your ACCELERATOR_TYPE and MODEL_SERVER combination, source prints an error that names the supported engines and stops. Read that message rather than carrying on: a later kubectl apply -k would just fail on a missing directory.

4. Install the CRDs, namespace and token Secret

Install the Gateway API Inference Extension CRDs. GAIE_URL is computed by env.sh (on this branch it resolves to releases/download/v1.6.2):

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/${GAIE_URL}/v1-manifests.yaml

Create the namespace and the Hugging Face Secret. Both commands are idempotent, so you can re-run them safely:

kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic llm-d-hf-token \
  --from-literal="HF_TOKEN=${HF_TOKEN}" \
  --namespace "${NAMESPACE}" \
  --dry-run=client -o yaml | kubectl apply -f -

The Secret name and key matter. The vLLM patch reads HF_TOKEN from a Secret called exactly llm-d-hf-token, so a differently named Secret leaves the pods unable to download the model.

5. Deploy the llm-d Router

Point Helm at the shared base values and the guide’s own values:

export ROUTER_BASE_VALUES="${REPO_ROOT}/guides/recipes/router/base.values.yaml"
export ROUTER_VALUES="${REPO_ROOT}/guides/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml"

(For TensorRT-LLM the second file is ${GUIDE_NAME}-trtllm.values.yaml instead.)

Then install the router in Standalone Mode, the default, which runs the EPP with an Envoy sidecar and needs no separate Gateway:

helm install ${GUIDE_NAME} \
  ${ROUTER_STANDALONE_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  ${MONITORING_VALUES} \
  -f ${ROUTER_VALUES} \
  ${ACCELERATOR_VALUES:-} \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

The guide’s values file shows what the router is configured to do. Note the new v1 API version for the plugin config:

apiVersion: llm-d.ai/v1
kind: EndpointPickerConfig
plugins:
- type: approx-prefix-cache-producer
- type: inflight-load-producer
- type: prefix-cache-affinity-filter
- type: token-load-scorer
schedulingProfiles:
- name: default
  plugins:
  - pluginRef: prefix-cache-affinity-filter
  - pluginRef: token-load-scorer

The router finds its model servers by the label llm-d.ai/guide: "optimized-baseline". If you later write your own Deployment, it needs that label or the router will have no endpoints.

Gateway Mode instead

If you already run a Kubernetes Gateway (GKE, Istio or agentgateway), deploy a Gateway named llm-d-inference-gateway first, then use the gateway chart with an HTTPRoute:

export GW_PROVIDER_NAME=none # options: none, gke, agentgateway, istio
helm install ${GUIDE_NAME} \
  ${ROUTER_GATEWAY_CHART} \
  -f ${ROUTER_BASE_VALUES} \
  ${MONITORING_VALUES} \
  -f ${ROUTER_VALUES} \
  ${ACCELERATOR_VALUES:-} \
  --set provider.name=${GW_PROVIDER_NAME} \
  --set httpRoute.create=true \
  --set httpRoute.inferenceGatewayName=llm-d-inference-gateway \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

Gotcha: set GW_PROVIDER_NAME to your real provider. The guide warns that the default none renders no provider-specific resources. On GKE that means no HealthCheckPolicy, so the Gateway marks the backends unhealthy and every request fails with a 503.

6. Deploy the vLLM model servers

One command applies the overlay for your accelerator, engine and infrastructure provider:

kubectl apply -n ${NAMESPACE} \
  -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}/

For NVIDIA GPUs and vLLM, the base overlay creates a Deployment with replicas: 2 that runs:

vllm serve Qwen/Qwen3-32B \
  --disable-access-log-for-endpoints=/health,/metrics,/v1/models \
  --tensor-parallel-size=2 \
  --enable-auto-tool-choice \
  --tool-call-parser=hermes \
  --reasoning-parser=qwen3 \
  --shutdown-timeout=45

Each pod requests 2 GPUs, 8 CPUs and 96 GiB of memory (limits: 16 CPUs, 128 GiB) and mounts a 20 GiB in-memory /dev/shm. The startup probe allows up to 120 failures 30 seconds apart, so a first start that is still downloading 32B weights is not killed early. Watch the pods come up:

kubectl get pods -n ${NAMESPACE} -w

Running on fewer GPUs

The guide says that if your cluster has fewer accelerators, you should adjust replicas and --tensor-parallel-size in the model server patch: guides/optimized-baseline/modelserver/gpu/vllm/base/patch-vllm.yaml. Two things to keep in mind:

• Change the nvidia.com/gpu request and limit to match the new tensor-parallel size.
• Prefix-cache routing only has something to choose between with two or more replicas. With one replica you get a working endpoint but none of the routing benefit.

Graceful draining is built in

The NVIDIA, AMD, CPU, Intel XPU and TPU vLLM overlays drain in-flight requests on rollouts and scale-downs. They combine vLLM’s --shutdown-timeout=45 with a 15-second preStop sleep and a 75-second termination grace period. If you raise one, raise the others with it, as the comment in the patch file says.

7. Send a test request

In Standalone Mode the endpoint is the router’s -epp Service:

export IP=$(kubectl get service ${GUIDE_NAME}-epp -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')

(In Gateway Mode: kubectl get gateway llm-d-inference-gateway -n ${NAMESPACE} -o jsonpath='{.status.addresses[0].value}'.)

That is a ClusterIP, so test from a temporary pod inside the cluster:

kubectl run curl-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'curl -sS -X POST "http://${IP}/v1/completions" -H "Content-Type: application/json" -d "{\"model\": \"${MODEL}\", \"prompt\": \"How are you today?\"}"'

A JSON completion back means the router, the Envoy sidecar and at least one vLLM pod are all working. It does not yet prove the routing is smart.

8. Prove prefix-cache routing is working

This is the step most quick-start posts skip. The official guide sends 10 requests that share a prompt prefix of about 2,000 tokens and differ only in the last sentence:

kubectl run prefix-test --rm -i --restart=Never \
  --image=${CURL_TEST_IMAGE} \
  --namespace="${NAMESPACE}" \
  --env="IP=${IP}" \
  --env="MODEL=${MODEL}" \
  -- /bin/sh -c 'P=$(for i in $(seq 1 100); do printf "The llm-d router keeps a conversation on the server that already caches its prefix. "; done)
    for i in $(seq 1 10); do
      curl -sS -o /dev/null -w "request ${i}: HTTP %{http_code}\n" -X POST "http://${IP}/v1/completions" \
        -H "Content-Type: application/json" \
        -d "{\"model\": \"${MODEL}\", \"prompt\": \"${P} Question ${i}: what is llm-d?\", \"max_tokens\": 16}"
    done'

Then read each pod’s counters through the Kubernetes API server proxy. You don’t need a port-forward:

for pod in $(kubectl get pods -n ${NAMESPACE} -l llm-d.ai/guide=${GUIDE_NAME} -o jsonpath='{.items[*].metadata.name}'); do
  echo "== ${pod}"
  kubectl get --raw "/api/v1/namespaces/${NAMESPACE}/pods/${pod}:8000/proxy/metrics" \
    | grep -E '^(vllm:prefix_cache_(hits|queries)_total|vllm:request_success_total|sglang:(cache_hit_rate|cached_tokens_total|num_requests_total))' || true
done

What a healthy result looks like with vLLM: one pod reports most of the 10 requests in vllm:request_success_total, and on that pod vllm:prefix_cache_hits_total is a large fraction of vllm:prefix_cache_queries_total, because every request after the first reuses the cached prefix. The other pod shows few or none of them.

With SGLang, look at sglang:num_requests_total and check that sglang:cached_tokens_total{cache_source="device"} goes up. Don’t rely on sglang:cache_hit_rate. The guide notes it is a rolling gauge that drops back to 0 after the burst on lmsysorg/sglang:v0.5.20, and it also reads 0 on the Intel XPU SGLang image even when hits occur.

If the 10 requests are spread evenly and hit rates stay near zero, prefix-cache affinity is not taking effect. Go to the troubleshooting section below.

9. Calibrate peakPrefillThroughput if you changed the model or hardware

This is the setting most likely to quietly hurt you after you leave the reference setup. The prefix-cache-affinity-filter uses peakPrefillThroughput to estimate each pod’s time-to-first-token from its in-flight load. That estimate decides when the saturation override kicks in and spreads traffic. The plugin default, 15928, was calibrated for Qwen3-32B on H100 80 GB with TP=2. The calibration guide notes that the Qwen3-Coder-480B guide ships 16444, measured for Qwen3-Coder-480B-FP8 on TPU v7x.

The repo includes a calibration script at guides/recipes/router/calibration/calibrate.sh. It runs a short Kubernetes Job that sends warmup and measurement requests of random token IDs, so every request misses the prefix cache and you measure true prefill speed. It then computes peakPrefillThroughput = CHUNK_SIZE / median(TTFT). It only prints the value and changes nothing:

GUIDE_NAME=optimized-baseline \
NAMESPACE=llm-d-optimized-baseline \
MODEL_NAME=Qwen/Qwen3-32B \
CHUNK_SIZE=8192 \
./calibrate.sh

Two details from the calibration docs:

• CHUNK_SIZE must match vLLM’s --max-num-batched-tokens. The default is 8192.
• It needs kubectl and envsubst on your PATH, and the stack must already be serving.

Put the measured number under parameters on the prefix-cache-affinity-filter plugin in your router values file, then helm upgrade the release. The repo’s configuration-matrix.md lists the reference values already measured for each model and accelerator, so check it before running your own.

10. Optional: turn on monitoring

If you run Prometheus (OSS or OpenShift), set MONITORING=true and add the router monitoring values before the helm install in step 5:

export MONITORING_VALUES="-f ${REPO_ROOT}/guides/recipes/router/features/monitoring.values.yaml"

On GKE with Managed Prometheus, use monitoring-gke.values.yaml instead. After the model servers are up, add the model-server PodMonitors:

kubectl apply -n ${NAMESPACE} -k ${REPO_ROOT}/guides/recipes/modelserver/components/monitoring

The guide’s advice is to watch load balance and cache hit rate together, because the two routing goals pull against each other. These are the signals it names for vLLM:

Signal	What it tells you
llm_d_epp_request_total, vllm:num_requests_running per pod	One hot pod next to idle ones means load balancing isn’t working
vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total	A falling ratio means requests aren’t landing on sticky pods
vllm:kv_cache_usage_perc, vllm:num_requests_waiting	These drive the saturation override
llm_d_epp_plugin_duration_seconds	Rising routing latency with healthy pods points at the router
vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds	The user-facing SLOs the setup protects

11. Troubleshooting

source env.sh exits with “does not support MODEL_SERVER=… on ACCELERATOR_TYPE=…”. The guide has no overlay for that pair. Use one of the engines the message lists, or check the support table in the guide’s README.

Requests through a GKE Gateway return 503. You left GW_PROVIDER_NAME=none. Re-install with gke so the chart renders the HealthCheckPolicy.

Pods never become ready, or crash while downloading weights. Check that the Secret is named llm-d-hf-token with key HF_TOKEN. On GKE TPU node pools, the guide also warns that the 100 GB default boot disk is too small: weights go to a hostPath cache, and the pod gets evicted mid-download. It recommends at least 200 GB for Qwen3-32B on v6e.

Load is uneven: some pods hot, others idle. The load-aware scorer or the saturation override isn’t spreading work. On non-default hardware this usually means peakPrefillThroughput is miscalibrated (step 9).

Prefix-cache hit rate is low. Either your prompts don’t share prefixes, or the override is spreading load so aggressively that it defeats affinity. The guide’s test: if pods are cold but hit rate is still low, the problem is the prompt pattern, not the override.

TTFT or ITL got worse even though load is balanced and the cache is healthy. Check llm_d_epp_plugin_duration_seconds and model-server queue depth before touching the routing config. The guide says the bottleneck is most likely the model servers, not the scheduler.

DRA overlays fail to apply on Kubernetes 1.32 or 1.33. They need the resource.k8s.io/v1 APIs from Kubernetes 1.34+. The driver may install on older versions, but the overlays still fail.

12. Upgrading from llm-d 0.x: what changes

If you already run llm-d 0.11 or earlier, read these before re-applying anything. All come from the v1.0.0 release notes:

• Version alignment. All llm-d components jumped to v1.0.0; for example the router endpoint picker moved from v0.11.0. The project says it will keep major and minor versions aligned from now on, so match versions across components rather than mixing.
• Router config is now llm-d.ai/v1. The configuration API graduated to v1, and flow control settings were consolidated under flowControl. The announcement says the flow-control defaults still behave like first-come-first-served, so nothing changes until you opt into multi-tenant policy.
• The ROCm image is deprecated. ghcr.io/llm-d/llm-d-rocm builds stopped. For prefill/decode on AMD, this release cherry-picked vLLM’s v0.32.0-rc.2 ROCm image into ghcr.io/llm-d/vllm-openai-rocm:v0.32.0-rc.2 as a temporary measure.
• Engine bumps. The default vLLM image went from v0.26.0 to v0.30.0, and SGLang from v0.5.20 to v0.5.22. Re-test any vLLM flags you added yourself.
• Prefill/decode needs a newer LeaderWorkerSet. The PD-disaggregation guide now needs LWS v0.11.1 or newer with DisaggregatedSet enabled (--set enableDisaggregatedSet=true in Helm). The release notes say almost all guides will break without v0.11.1+ and strongly recommend LWS v1.0.0.
• llm-d-kv-cache moved into llm-d-router. It is kept around while users migrate.

13. Clean up

kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/${GUIDE_NAME}/modelserver/${ACCELERATOR_TYPE}/${MODEL_SERVER}/${INFRA_PROVIDER}
helm uninstall ${GUIDE_NAME} -n ${NAMESPACE}
kubectl delete -n ${NAMESPACE} -k ${REPO_ROOT}/guides/recipes/modelserver/components/monitoring --ignore-not-found=true
kubectl delete namespace ${NAMESPACE}

Where to go next

The optimized baseline is the starting point the other llm-d guides build on. From the same guides/ directory on release-1.0:

• Precise prefix cache routing replaces the approximate prefix estimate with a global index of each vLLM server’s real KV-cache state.
• Tiered prefix cache offloads KV cache to CPU or disk for long multi-turn sessions.
• Prefill/decode disaggregation and wide expert parallelism are for large models such as openai/gpt-oss-120b and deepseek-ai/DeepSeek-R1.
• Flow control is for multi-tenant priority and fairness, the headline 1.0 feature.

If your bottleneck is a single server reusing its own cache badly rather than routing across servers, start with our guide to fixing KV cache reuse in llama-server. For a hosted model with no infrastructure at all, see how to use Gemini 3.6 Flash.

FAQ

Is llm-d a model server? No. It is a routing and scheduling layer in front of model servers like vLLM and SGLang. In 1.0 the project stopped building most of its own inference images and points at upstream engine images instead.

Can I try it without GPUs? The guide includes a community-maintained cpu overlay serving meta-llama/Llama-3.2-3B-Instruct. It needs x86 CPUs with AMX or AVX512-BF16 (Sapphire Rapids or newer, GCP C3, AMD Zen 4+) and reserves 64 cores and 64 GiB per replica across 2 replicas. CPUs without those instructions need --dtype=float32.

Standalone or Gateway Mode? Standalone is the default and what every other guide uses. Pick Gateway Mode only if you already run a Kubernetes Gateway and want the router behind it.

Does it speak the OpenAI API? Yes. The test requests above hit /v1/completions, and 1.0 adds previous_response_id to its Responses API support.