llm-d vs Dynamo vs vLLM Production Stack: Which to Run

Both projects that define this layer define it the same way. Dynamo's README calls itself an orchestration layer above inference engines and says it does not replace SGLang, TensorRT-LLM or vLLM; llm-d's says model servers run models on accelerators and llm-d orchestrates above them. production-stack is the same layer on a narrower base: one engine.

This is replica selection inside one model pool by cache overlap, not routing between models or providers. Which provider or price tier a request should go to is the AI gateway decision and routing cheap models first, a different layer with a different buyer. This post also starts after the engine choice, which is SGLang versus vLLM and vLLM versus Ollama versus TensorRT-LLM.

Prefill and decode disaggregation no longer separates them. All three ship it: llm-d has a pd-disaggregation guide, Dynamo marks disaggregated serving supported on all three of its backends, and production-stack has two disaggregated-prefill routers. Whether to split the pools belongs to prefill and decode disaggregation at serving scale.

Three and not five, deliberately. KServe v0.20.0, published 6 August 2026, upgraded its llm-d components and its Gateway API Inference Extension dependency, so its LLM path consumes llm-d rather than competing with it. AIBrix is a real fourth option, Apache-2.0, under the vLLM organisation, released v0.7.0 on 18 June 2026, with a white paper at arXiv:2504.03648. We read its feature list, not its code, so it gets no table row.

The real difference is not which algorithms exist but the shape each project makes you express the decision in: a plugin graph, an optional enum and a required enum.

production-stack takes --routing-logic, required, with no default and six values at the shipped tag. Both of its README files understate this, one listing three algorithms and the other only roundrobin and session. The parser is the authority, and the spellings are literal: prefixaware is one word, the two disaggregated values use underscores, and the KV-aware value is kvaware. A hyphen in any of them fails at startup.

vllm-router --port 8001 \
    --service-discovery k8s \
    --k8s-namespace vllm \
    --k8s-label-selector environment=production \
    --routing-logic prefixaware \
    --prefix-min-match-length 512 \
    --engine-stats-interval 10 \
    --log-format json

That --prefix-min-match-length 512 is not decoration: the flag defaults to 0, which disables the threshold, and below it the router falls back to QPS-based routing. The prefix-aware router's docstring is honest about its model of the world: it assumes no eviction of the prefix cache. kvaware avoids that assumption by asking the engines and pays a dependency for it, building an LMCache controller manager over ZMQ sockets, with --kv-aware-threshold defaulting to 2000.

Dynamo puts the same decision behind --router-mode, or DYN_ROUTER_MODE, with seven values, of which kv is the cache-aware one. The KV path is then tuned numerically rather than structurally: --router-kv-overlap-score-credit defaults to 1.0 and at 0 the router ignores prefix caches and skips creating a local indexer, and host and disk cache hits weigh 0.75 against 0.25. Selecting the mode is one flag on the frontend, plus KV event publishing on each worker:

export PYTHONHASHSEED=0
python3 -m dynamo.frontend --router-mode kv &

DYN_SYSTEM_PORT=8081 CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm \
    --model Qwen/Qwen3-0.6B \
    --block-size 64 \
    --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20080","enable_kv_cache_events":true}' &

DYN_SYSTEM_PORT=8082 CUDA_VISIBLE_DEVICES=1 python3 -m dynamo.vllm \
    --model Qwen/Qwen3-0.6B \
    --block-size 64 \
    --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20081","enable_kv_cache_events":true}' &

llm-d has no mode enum at all. Routing is a document: an EndpointPickerConfig at llm-d.ai/v1alpha1, with a plugins list and schedulingProfiles that reference plugins by pluginRef. The optimized-baseline guide composes four.

apiVersion: llm-d.ai/v1alpha1
kind: EndpointPickerConfig
plugins:
- type: approx-prefix-cache-producer
- type: inflight-load-producer
- type: prefix-cache-affinity-filter
- type: token-load-scorer
schedulingProfiles:
- name: default
  plugins:
  - pluginRef: prefix-cache-affinity-filter
  - pluginRef: token-load-scorer

One tag, two vocabularies. The no-Kubernetes guide in the same release writes an EndpointPickerConfig with entirely different plugin names for the same job: file-discovery, then queue-scorer, kv-cache-utilization-scorer, prefix-cache-scorer and no-hit-lru-scorer at weights 2, 2, 3 and 2. Its own comment claims that set mirrors the optimized baseline, which it does not. Treat them as two configurations to choose between, not as old and new.

The tunables carry hardware assumptions, and llm-d says so: peakPrefillThroughput 15928 on prefix-cache-affinity-filter is calibrated for Qwen3-32B on an H100 80 GB at tensor parallel 2, and the guide tells you to measure your own.

None of this is the engine's own prefix cache, a separate mechanism covered in why prefix cache hit rates collapse in multi-turn agents. This layer decides which replica a request reaches, so that the cache on that replica is warm for it. A perfect engine cache behind round-robin selection is a cache you pay for and do not use.

Dynamo is the only one of the three that publishes a per-backend feature matrix, worth reading before you assume parity. It lists six features across SGLang, TensorRT-LLM and vLLM. Five of them, disaggregated serving, KV-aware routing, the SLA-based planner, multimodal and tool calling, are supported on all three. The sixth, KVBM, is marked in progress on SGLang and supported on the other two. Run two engines behind one layer, as the SGLang versus vLLM comparison argues is increasingly common, and that matrix tells you which capability disappears when a workload moves.

llm-d ships images for all three engines plus a new Intel XPU SGLang image, so it is not the vLLM-centric option. Its well-lit paths do lean vLLM: v0.9.0 carries 19 guide directories, 9 of them with committed benchmark-results, and workload-autoscaling alone holds six variants.

production-stack is vLLM-only by design and coupled to LMCache: its default images come from the lmcache organisation, and kvaware will not start without an LMCache controller. Our own Ollama versus vLLM comparison called this stack Helm charts, dashboards and model-aware routing out of the box, which is worth correcting: the routing is a strategy flag you must set, and the charts are unpinned.

Read the published numbers carefully. llm-d's README lists six performance results and Dynamo's lists five, most crediting an outside party on named hardware, but two of llm-d's six and one of Dynamo's five link only to the project's own blog. Treat the set as existence proofs of a mechanism, not as numbers you will hit. production-stack publishes no headline figure, while shipping a benchmark harness and two benchmarking tutorials.

llm-d and Dynamo have converged on the same Kubernetes standard: an endpoint picker behind a Gateway API Inference Extension gateway, with the pool exposed as an InferencePool on a standard HTTPRoute. This is the object both sit behind:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: qwen
spec:
  parentRefs:
  - name: inference-gateway
  rules:
  - matches:
    - headers:
      - name: X-Gateway-Model-Name
        type: Exact
        value: Qwen/Qwen3-0.6B
      path:
        type: PathPrefix
        value: /
    backendRefs:
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: qwen-pool
      port: 8000
    timeouts:
      request: 300s

llm-d v0.9.0 pins Gateway API CRDs at v1.5.1 and the inference extension at v1.5.0 in guides/env.sh. Upstream, the inference extension released v1.6.0 on 17 August 2026, the same day llm-d shipped v0.9.0, so llm-d is one minor behind by hours rather than by neglect. Dynamo documents two topologies: its native path, client to frontend to router to workers, and the gateway path, client to gateway to endpoint picker to a frontend sidecar running in --router-mode direct.

What ports is the wire path: the Gateway, the HTTPRoute, the InferencePool and the header-based model match. What does not port is the tuning. llm-d writes an EndpointPickerConfig with its own plugin type names; Dynamo writes an epp component with its own, a disagg-profile-handler and a label-filter keyed on nvidia.com/dynamo-component-type. Migrating means rewriting every scheduling decision you made, under a gateway that stays byte-identical. production-stack sits outside this convergence, with its own router process and its own operator CRD group.

A managed endpoint hides this whole layer, which is why the cloud buyer never chooses. A team with 8 to 32 GPUs in its own racks has to. Start with the API version, where all three answer differently.

Every manifest a platform team writes this quarter targets an alpha API or a group mid-migration. That belongs in the risk register at design time, not in the postmortem of the first upgrade. Dynamo's request kind is at least already served at v1beta1:

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: my-model
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  sla:
    ttft: 200.0   # ms
    itl: 20.0     # ms
  autoApply: true

Those two SLA numbers drive Dynamo's planner, whose own defaults are an on-prem shape check: ttft_ms 500.0, itl_ms 50.0, max_gpu_budget 8, min_endpoint 1. A default GPU budget of 8 is the size of cluster this argument is about. Do not accept 500 ms and 50 ms without checking them against your workload; TTFT and tokens-per-second targets by workload is the basis for setting them.

Then the cluster question. Dynamo runs on a single node with --discovery-backend file and needs neither etcd nor NATS. llm-d has a no-Kubernetes path too, deploying the endpoint picker, Envoy and vLLM workers as plain YAML, with the picker reading /etc/epp/endpoints.yaml, where every entry must be a literal IPv4 address because file-discovery does not resolve hostnames. production-stack has no such path: it is a Helm chart.

The awkward part for llm-d is that the route an air-gapped team would take is pinned furthest behind the release. That same guide exports VLLM_IMAGE as vllm/vllm-openai:v0.19.1 while the rest of the v0.9.0 release ships v0.26.0, and it uses the second plugin vocabulary. Nothing explains the gap, so verify that floor yourself.

None of the three documents an air-gapped install procedure, so the mirror list is something you assemble. llm-d pulls charts and its endpoint picker image from ghcr.io, engine images from docker.io and the TensorRT-LLM image from nvcr.io. Dynamo pulls its platform chart from an NGC Helm host and its runtimes from nvcr.io. production-stack pulls its chart from a GitHub Pages repository, its images from the lmcache organisation, and two Prometheus subcharts from an OCI registry. Node-level egress is covered in air-gapped vLLM deployment; this layer multiplies the registries a change window has to escort.

Run Dynamo if your accelerators are NVIDIA, you want one artifact to qualify, and you value that a patch release does not move the engine underneath you. It is also the cheapest of the three to pilot, since it needs no cluster.

Run llm-d if you need AMD, Intel XPU, TPU or CPU serving, or if you want committed reproducible recipes per accelerator rather than a matrix in a README. Budget for an engine re-qualification per minor release, and for a routing configuration that is alpha-versioned and composed rather than selected.

Run the vLLM production stack if you are vLLM-only, already run LMCache, and want the smallest thing that works. It is the shortest install of the three, and the only one whose defaults will hurt you in a validated environment, so pin the four image tags and the three pull policies first.

For a regulated buyer, score them in this order: engine pinning first, CRD API stability second, registry surface third, routing sophistication fourth. Throughput claims do not enter the ranking, because no published figure at this layer was measured on your hardware. The rest of the framework for this cluster sits on our LLM models pillar.

This week, render whichever one you are trialling and read the images out of it. For the Helm chart that is helm template vllm vllm/vllm-stack -f your-values.yaml | grep -E 'image:'; for llm-d it is kustomize build on the overlay your guide points at. Diff that list against the engine version your validation evidence describes. If any line ends in :latest, you have found the thing that will move underneath you, before the auditor did.