Topic hub
LLMs & Models
Understand large language models, fine-tuning, prompt engineering, and model selection.
All posts · 58
LLMS & MODELS01
vLLM Multi-LoRA Serving: Adapter Limits and Silent FailuresvLLM serves one LoRA adapter per batch by default, and in v0.32.0 an adapter that applies to zero layers is still reported only by a DEBUG log line.
LLMS & MODELS02
vLLM CPU Offload Not Working: Fix --cpu-offload-gb (2026)At vLLM 0.31, --cpu-offload-gb works on both model runners. When it fails, it offloads nothing, the wrong weights, or costs up to 1.9x the host RAM.
LLMS & MODELS03
vLLM Structured Output Not Working: HTTP 200, No JSONvLLM removed guided_json in v0.12.0 but still answers it with HTTP 200 and free text. Ten failure modes stamped by version, their fixes, and a CI canary.
LLMS & MODELS04
No User Query Found in Messages: Qwen Tool Loop FixQwen 3.5, 3.6 and 3.8 templates reject any request with zero plain user turns. On Ollama qwen3.8, a tool loop outgrowing num_ctx gets there as a 500.
LLMS & MODELS05
Expert Parallelism vs Tensor Parallelism for MoE ServingvLLM puts MoE experts in a tensor-parallel group of size DP x TP unless you pass one flag. EP size is derived as TP x DP and cannot be set directly.
LLMS & MODELS06
KV Cache Duplication Factor: Divide tp_size by KV HeadsYour KV cache is duplicated max(1, tp_size // H) times. At 8 kv-heads and -tp 8 that is 1. DeepSeek-R1's config.json says 128 and the real H is 1.
LLMS & MODELS07
vLLM rope_scaling YaRN: What 128k Context Really CostsvLLM deleted the --rope-scaling flag in November 2025. The rope_parameters override that parses on v0.28.0, and what factor 4.0 charges every short prompt.
LLMS & MODELS08
Ollama num_ctx: Why Long Prompts Get Silently TruncatedOllama's default context window has been 262144, 32768 or 4096 since February 2026, picked from detected VRAM. Two truncation layers behave in opposite ways.
LLMS & MODELS09
llm-d vs Dynamo vs vLLM Production Stack: Which to Runllm-d v0.9.0 moved vLLM three minor versions in one release. Dynamo v1.4.1 moved none. The orchestration layer you pick is a change-control posture.