vLLM KV Cache OOM: The Flags That Actually Fix It

vLLM's memory profiler does the accounting explicitly. Available KV cache memory is the requested memory (total GPU memory multiplied by gpu_memory_utilization) minus non-KV-cache memory, minus the CUDA graph estimate. Non-KV-cache memory is itself a sum of four terms: model weights, peak activation memory measured during a profiling forward pass, non-torch allocations, and CUDA graph memory. The profiler subtracts a small redundancy buffer (150 MiB) on top, because profiling has been observed to slightly underestimate real consumption.

Read that as a budget with five claimants:

vLLM logs its result as Available KV cache memory: X GiB. That line is the single most useful thing in the startup output and most people scroll past it. If that number is smaller than you expect, the equation tells you which term to attack. If it is negative or near zero, you get the first error above.

The demand side is arithmetic you can do yourself. Per token, per sequence, KV cache costs:

bytes_per_token = 2 * num_layers * num_kv_heads * head_dim * dtype_bytes

The leading 2 covers keys and values. Under grouped-query attention, num_kv_heads is much smaller than the attention head count, which is the main reason modern models are cheaper to serve than their parameter count suggests. Take a typical 8B model with 32 layers, 8 KV heads, and a head dimension of 128, at fp16:

2 * 32 * 8 * 128 * 2 = 131,072 bytes = 128 KiB per token

At a 128k context, one sequence holds 16 GiB of KV cache. That is roughly what the weights themselves occupy at fp16. One request at full context can cost as much as the entire model, which is the whole reason --max-model-len is the highest-leverage flag on the list: it multiplies directly into your concurrency ceiling, and the default is whatever the model config claims rather than what you actually serve.

Models using multi-head latent attention compress K and V into a shared low-rank latent instead of storing per-head tensors, so the formula above overstates them substantially. If you are serving one, measure rather than extrapolate.

The second error is the more common one and the easier fix, because the ratio in the message is diagnostic. If the model declares 131,072 tokens and the cache holds 30,880, you are not short by a rounding error. You are serving a context window you almost certainly do not need. Setting --max-model-len 16384 when your P99 prompt is 6k tokens is not a compromise, it is correcting a default.

The first error deserves one specific check before anything else. Run nvidia-smi and confirm nothing else holds memory on the device. vLLM's profiling snapshot is taken once, at startup, against free memory at that instant. A leftover process, a notebook kernel, or a second vLLM instance sharing the card all corrupt the measurement, and the resulting failure looks like a vLLM sizing problem rather than a housekeeping one. Dedicating a GPU to a single vLLM process is the documented recommendation and it removes an entire class of confusing behavior.

If a configuration that worked for months started failing after an upgrade, this is almost certainly why: CUDA graph memory profiling became the default in v0.21.0, and CUDA graph capture consumes more memory in the V1 engine than it did in V0.

Before that change, graph memory was not subtracted before KV cache allocation. Now it is. The same --gpu-memory-utilization value therefore yields a smaller KV cache than it used to, with no config change on your side. vLLM is unusually helpful about this and logs the translation directly, telling you what your current value is equivalent to under the old behavior and what to raise it to for the same effective cache size.

That log line is the answer. Follow it rather than reverse-engineering the difference. The escape hatch, VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0, restores the old accounting, and vLLM warns that without it graph memory is unaccounted during allocation, which may require lowering utilization to avoid OOM. That trade is bad in production: you are moving a deterministic startup failure into a nondeterministic runtime one.

The blunter version of the same lever is --enforce-eager, which skips graph capture entirely. It frees real memory and costs steady-state decode performance, which makes it a good development default and a poor production one. In between sits cudagraph_capture_sizes inside compilation_config, which by default captures batch sizes up to max_num_seqs. Capturing fewer sizes reclaims memory while keeping graphs for the batch sizes you actually hit.

--gpu-memory-utilization is a fraction, which means the KV cache you end up with is a residual: whatever survives after four other terms. That is a strange way to control the thing you care about most.

--kv-cache-memory-bytes inverts it. Set it and it overrides gpu_memory_utilization for cache sizing, pinning KV cache to a value you chose. vLLM's profiler even computes a suggested optimal figure during startup, derived from free memory minus model, activation, non-torch, and graph memory, minus the redundancy buffer.

The workflow that follows is simple and worth adopting as standard practice:

Step four is the one that matters six months out. A pinned number is a config you can review. A ratio is a number that means something different on every card, every model, and every upgrade.

--kv-cache-dtype defaults to auto, which means the cache inherits the model's dtype. Setting it to fp8 stores K and V in 8 bits and runs attention in fp8 e4m3, which roughly halves KV storage.

The current numbers, from vLLM's own April 2026 analysis, are specific enough to plan against:

Testing was on H100 and B200 class hardware with current attention backends. The break-even line is the part most posts omit: below roughly 7k tokens of context, the quantization overhead dominates and you lose. fp8 KV cache is a long-context optimization, not a universal one.

Skip it when your contexts are short, when the model uses a head dimension of 256 on prefill-heavy traffic (a 1.6x time-to-first-token slowdown was measured), or when uncalibrated accuracy on your own evals drops below 95%. That last condition is the only one that matters, and it is why this belongs in your evaluation harness rather than in a config review. The wider point about measuring model changes on your own traffic rather than published benchmarks is one we make in our guide to running GLM-5.2 locally as well.

**VLLM_ALLOW_LONG_MAX_MODEL_LEN=1.** This overrides the safety check that keeps max_model_len within what the model's config declares. The documentation is explicit that exceeding the derived length may produce incorrect model outputs or CUDA errors. It exists for models whose config understates their real trained length. Reaching for it to squeeze out more context is how you get garbage generations and a crash that looks like a memory bug.

**Expecting --swap-space to save you.** In V1 the default preemption mode is recompute, not swap. When the scheduler runs out of KV blocks it evicts and recomputes rather than paging to host memory. Swap space is not the relief valve it appeared to be under V0. If preemption appears in your logs, the documented remedies are more cache (higher utilization, larger tensor parallel size) or less demand (lower max_num_seqs, lower max_model_len).

Confusing weight pressure with cache pressure. --cpu-offload-gb (default 0) offloads model parameters to host RAM, not KV cache. It is the right tool when weights do not fit and the wrong one when the cache is undersized, and the speed cost is significant either way. Diagnose which term of the equation is actually short before choosing.

Tensor parallel imbalance. With --tensor-parallel-size above 1, every rank profiles independently against its own card. A single GPU carrying extra load, a display output, or a stray process becomes the binding constraint for the whole deployment, and the error surfaces from whichever rank lost. Verify all ranks are on clean devices before tuning anything.

Prefix caching, which is a cache and behaves like one. vLLM identifies each KV block by hashing its tokens together with the parent hash, plus LoRA IDs, multimodal input hashes, and cache salts. Blocks with a live reference count are not freed on eviction, they are only removed from the hash table, so cached prefixes hold real memory while in use. It is a large throughput win on shared system prompts and a source of memory pressure you did not schedule for.

That cache salt field is worth knowing about if you serve more than one customer from one engine, since it is the mechanism that keeps one tenant's cached prefix from being served to another. It is the same class of boundary we cover in multi-tenant RAG isolation: a shared structure with a tenant discriminator, where the discriminator is the entire security property.

Multimodal caches you forgot about. mm_processor_cache_gb defaults to 4 GiB and VLLM_CPU_KVCACHE_SPACE defaults to 4 GiB. Neither is large, both are real, and on a constrained card they are free memory if you are not serving images.

When an OOM appears, work the equation from cheapest to most invasive:

Notice that raising --gpu-memory-utilization is not on that list until it is genuinely the answer, which is when the card is dedicated, the log confirms the CUDA graph accounting change, and vLLM has told you the specific value to move to.

Memory sizing is the part of self-hosting that decides whether the economics work, because your concurrency ceiling is what the cost model divides by. If you are still deciding whether to self-host at all, our break-even analysis of self-hosting versus API covers the arithmetic, and our comparison of vLLM, Ollama, and TensorRT-LLM covers picking the engine. For pushing past what one engine can do on one card, prefill-decode disaggregation is the next architectural step, and SGLang versus vLLM covers the closest alternative. More on model selection and serving sits in our LLMs and models pillar.

The summary is that vLLM is not being mysterious. It publishes the equation, logs its result, and tells you which term to change. The failure mode is that its error messages suggest the one flag that is usually wrong, and most teams take the suggestion.