vLLM CPU Offload Not Working: Fix --cpu-offload-gb (2026)

There are now two offload backends, chosen by --offload-backend (auto, uva or prefetch, default auto):

Under auto, vLLM picks prefetch if --offload-group-size is above 0, otherwise UVA if --cpu-offload-gb is above 0, otherwise a NoopOffloader that does nothing.

The config docstring gives the intuition: one 24 GB GPU with cpu_offload_gb=10 behaves like a virtual 34 GB GPU, enough for a 13B BF16 model that needs at least 26 GB. The same docstring states the condition that decides performance: this "requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass."

Two scope limits matter for everything that follows. First, offload is applied inside make_layers(), the helper that builds the decoder layer stack, and to multimodal towers on the UVA backend only. Token embeddings and lm_head are built outside it and never move, and a model that neither builds its stack with make_layers() nor calls the offloader directly (as Whisper's encoder does) gets no offload at all (llama, qwen2, qwen3_moe, deepseek_v2, gpt_oss, mixtral, glm4_moe, qwen3_next and gemma3 all use it at v0.31.0). Second, offload moves weights, not KV cache. The VRAM it frees can go to cache, but if your error is about cache blocks, the fix is in KV cache budgeting and gpu_memory_utilization, not here.

Run the server with a budget and grep three lines out of the log:

pip show vllm | grep Version

vllm serve /models/your-model --cpu-offload-gb 12 2>&1 | tee serve.log

# In a second shell, once weights have loaded:
grep -E "Offloader set to|Total CPU offloaded parameters|Model loading took" serve.log

What each line means at v0.31.0:

The prefetch backend logs Offloader set to PrefetchOffloader but never the Total CPU offloaded parameters line. It logs [PrefetchOffloader] Initialized N modules. Total GPU memory saved: X GB instead, and that GB is 10^9 bytes, not GiB.

If the version is 0.27.x or older, stop here and upgrade. PR 51413 added weight offloading to Model Runner V2 and merged on 2026-08-07; v0.27.0 and v0.27.1 do not contain it and v0.28.0 does. At v0.27.1, V2 was the default for generate models that are not MoE, not hybrid and not attention-free, plus a short MoE allow-list, so it was mainly dense models whose flag was silently dropped. The reporter of vLLM issue 56283 saw identical OOMs on a 16 GB RTX 5070 Ti with QwQ-32B-AWQ (19.33 GB) at every offload size on 0.27.1; on 0.29.0, same box and model, --cpu-offload-gb 8 reached healthy and logged 8.02 GiB offloaded.

On an air-gapped server that matters more than it sounds: moving from 0.27.x to 0.28.0 or later is a packaging and validation item with its own change ticket, not a pip install -U.

The table covers the version bug plus the six failures that remain at v0.31.0, in the order you should check them.

Row 2 deserves a note on the error you get if you go the other way. --offload-backend prefetch without --offload-group-size fails validation with offload_num_in_group (1) must be <= offload_group_size (0), because the group size defaults to 0 and the per-group count defaults to 1. That one is loud. The auto case is not: set --cpu-offload-gb 12 and --offload-group-size 8 together and vLLM emits a Python warning that the prefetch backend will be selected, then discards your 12 GiB.

Row 3 has a Python-only variant. The LLM() constructor at v0.31.0 has named arguments for cpu_offload_gb and for the prefetch fields, including offload_params, but no named cpu_offload_params. LLM(cpu_offload_gb=12, offload_params={"experts"}) therefore picks UVA from the budget and offloads non-selectively, because the UVA backend never reads offload_params. Pass cpu_offload_params explicitly; LLM(**kwargs) forwards it to EngineArgs, which has that field.

Selector matching is literal. The UVA offloader tests f".{param}." in f".{prefix}{name}.", and make_layers() passes no prefix, so name is relative to one decoder layer: self_attn.qkv_proj.weight, mlp.experts.w13_weight. experts, self_attn and mlp match. layers.0, model.layers.3.mlp.experts, embed_tokens and lm_head match nothing. A contributor matrix on vLLM issue 53931 (with --cpu-offload-gb 1.0) shows exactly this: embed_tokens and layers.0 produced no offload line and full GPU residency, self_attn offloaded 0.08 GiB, and no selector offloaded 0.67 GiB.

Without a selector, the UVA offloader walks named_parameters() in declaration order, layer by layer, until the budget runs out. On a sparse MoE, that means the attention projections, router, norms and shared expert of the first layers go to host memory, and every one of those is read on every token. The experts, most of which sit idle for any given token, stay on the GPU.

vLLM issue 54593 puts numbers on it for a sparse MoE with 512 experts and top-10 routing: dense weights (attention, router, norms, shared expert) were 4.80 GiB on disk, all of it read per token. Experts were 62.11 GiB on disk, of which 1.22 GiB was read per token. The default policy offloads from the 4.80 GiB side first. A fix that prioritizes experts was proposed upstream, but at v0.31.0 declaration order is still the default.

Restrict offload to the expert tensors:

vllm serve /models/your-moe-model \
  --offload-backend uva \
  --cpu-offload-gb 40 \
  --cpu-offload-params experts

experts matches mlp.experts.w13_weight, mlp.experts.w2_weight and their scale tensors. Set the budget to the gap between the model's weight footprint and your VRAM, plus headroom for KV cache. Two cautions before you trust it:

If a second GPU is available, splitting experts across cards keeps them in VRAM instead of streaming them over PCIe; the trade-offs are in expert parallelism versus tensor parallelism for MoE. And size first, offload second: GPU sizing for a large MoE with MXFP4 in vLLM shows how far quantized weights alone move the line.

With pinning on (the default on CUDA), each offloaded tensor goes through Tensor.pin_memory(), which allocates from PyTorch's caching host allocator. That allocator rounds each request up to the next power of two. The vLLM log reports the bytes it asked for, not what the allocator handed back.

vLLM issue 56410 measured this on v0.29.0. Per-tensor ratios for QwQ-32B-AWQ were 1.90 for gate_up_proj.qweight and down_proj.qweight, 1.83 for qkv_proj.qweight and 1.28 for o_proj.qweight; the same report gives 1.18 for Llama 3.3 70B AWQ and 1.15 for an 8B FP16 model. End to end, --cpu-offload-gb 4 (4.29 GB) took 8.01 GB of engine shared memory, 1.87x the request. The overhead depends on tensor shapes, from about 1.15x to roughly 1.9x.

At scale the rounding kills the process. On vLLM issue 58178, a reporter asked for 305 GiB of offload on an NVIDIA DGX Station GB300 with 494 GiB of host memory; about 504 GiB was consumed and the kernel OOM-killed the engine. Per layer, a 4.5 GiB w13_weight became 8 GiB pinned and a 2.25 GiB w2_weight became 4 GiB. Nothing in the vLLM log pointed at offload.

The workaround with a verified code path is one environment variable:

export VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1
vllm serve /models/your-moe-model \
  --offload-backend uva \
  --cpu-offload-gb 40 \
  --cpu-offload-params experts

# In a second shell, while weights load, read the engine's host memory:
PID=$(nvidia-smi --query-compute-apps=pid --format=csv,noheader | head -n1)
grep -E "VmRSS|RssShmem" /proc/$PID/status

With pinning disabled, the UVA path stays on, and vLLM's CUDA view helper allocates mapped host memory with cudaHostAlloc for exactly the tensor's size instead of going through the caching allocator. Issue 56410 measured 4.34 GB for the 4.29 GB request, 1.01x. On issue 58178 the same variable let the reporter's offload load with 126 GiB of host memory to spare.

Two limits. This is for the UVA path on CUDA GPUs (the issue 58178 reporter used it on a Grace-based GB300); do not combine it with VLLM_WEIGHT_OFFLOADING_DISABLE_UVA=1, because the non-UVA fallback then copies unpinned weights to the GPU on every forward call. And on unified-memory systems such as GH200, vLLM's own source notes that pinned memory eats into GPU memory, which is part of why this override exists.

An allocator-level alternative is reported in issue 58178 comments: on torch 2.13, which v0.31.0 pins, PYTORCH_CUDA_ALLOC_CONF=pinned_max_cached_size_mb:1024 allocates tensors above the limit at exact size, and one 47.7 GiB tensor took 48.6 GiB of host memory instead of 65.2 GiB. Torch 2.12.1 rejects the key, the setting is process-wide, and it slows re-pinning of large freed buffers. The same thread warns against pinned_max_round_threshold_mb on torch 2.13.0. Prefer the vLLM variable; its path is in the vLLM source.

Prefetch is the answer to "vllm offload layers to cpu" in the literal sense. It groups layers and offloads the last N of each group; the docstring's example is a group size of 8 with 2 per group, which offloads layers 6, 7, 14, 15, 22, 23 and so on. Copies back to the GPU run on a separate CUDA stream, one or more layers ahead, and the design targets torch.compile and CUDA graph capture.

vllm serve /models/your-model \
  --offload-backend prefetch \
  --offload-group-size 8 \
  --offload-num-in-group 2 \
  --offload-prefetch-step 1

Do not add --cpu-offload-gb to this command: with both backends' flags set, the explicit prefetch backend ignores the UVA settings with a warning, and under auto prefetch wins anyway. To restrict which tensors in the selected layers move, use --offload-params, not --cpu-offload-params; an empty selector moves every parameter of each selected layer. Raising --offload-prefetch-step hides more transfer latency at the cost of a larger static GPU buffer pool, which the init log line reports.

For a MoE where only the experts should leave the GPU, use UVA with --cpu-offload-params experts. For a dense model where every weight is read every token anyway, prefetch overlaps the copy with compute, and UVA offers no way to do that.

UVA reads offloaded bytes across the interconnect on every forward pass, so divide the bytes by the link's bandwidth to get a floor on step time. The L40S's PCIe Gen4 x16 link is specified at 64 GB/s bidirectional, roughly 32 GB/s per direction. The RTX PRO 6000 Server Edition runs PCIe Gen5 at roughly 64 GB/s per direction with no NVLink.

These are theoretical ceilings from arithmetic, not measurements; real zero-copy throughput sits below the link's rated figure, and GPU compute adds on top. Two consequences follow.

Batching amortizes the bus. One forward pass serves every sequence in the batch, so the per-step cost is shared, and aggregate throughput at 32 concurrent streams can be acceptable where a single stream is not.

Expert-only offload is cheap at batch 1 and gets more expensive with concurrency. With top-k routing, the set of distinct experts touched in one forward pass grows with batch size, so a busy server drifts toward reading the whole offloaded expert set every step. Research on expert-aware placement has been working this problem for years: offloading for fast MoE inference, Fiddler's CPU-GPU orchestration, MoE-Lightning and, for throughput-oriented offload generally, FlexGen. An expert GPU cache has been proposed for vLLM in RFC 38256, but no such flag ships in v0.31.0.

This is where on-prem changes the answer. In the cloud, a model slightly larger than VRAM is solved by renting a bigger instance. On a fixed L40S or RTX PRO 6000 box with a procurement cycle measured in quarters, offload is the lever you have, and a capacity plan can get signed off on the strength of a flag. Every failure above is silent at the flag level, so the plan has to be verified from the log lines and host memory before it goes to the change board, not discovered in production. And because the bus is the ceiling, the bandwidth table can tell you in advance that the honest answer is a quantized checkpoint, a second card, or a different engine.

That different engine is often llama.cpp. For a single-user workstation running one large MoE, its tensor-override expert offload is a better fit than a serving engine tuned for concurrency; the recipe is in running GLM-5.2 locally. For a multi-user service on regulated data, stay on vLLM and make offload a measured, budgeted part of the design. The wider set of model and serving decisions sits in our LLM models and inference guide.

Three checks on the box you already have:

If the model is a MoE, add --cpu-offload-params experts before you compare anything else.