Topic hub
LLMs & Models
Understand large language models, fine-tuning, prompt engineering, and model selection.
All posts · 56
LLMS & MODELS01
vLLM Structured Output Not Working: HTTP 200, No JSONvLLM removed guided_json in v0.12.0 but still answers it with HTTP 200 and free text. Ten failure modes stamped by version, their fixes, and a CI canary.
LLMS & MODELS02
No User Query Found in Messages: Qwen Tool Loop FixQwen 3.5, 3.6 and 3.8 templates reject any request with zero plain user turns. On Ollama qwen3.8, a tool loop outgrowing num_ctx gets there as a 500.
LLMS & MODELS03
Expert Parallelism vs Tensor Parallelism for MoE ServingvLLM puts MoE experts in a tensor-parallel group of size DP x TP unless you pass one flag. EP size is derived as TP x DP and cannot be set directly.
LLMS & MODELS04
KV Cache Duplication Factor: Divide tp_size by KV HeadsYour KV cache is duplicated max(1, tp_size // H) times. At 8 kv-heads and -tp 8 that is 1. DeepSeek-R1's config.json says 128 and the real H is 1.
LLMS & MODELS05
vLLM rope_scaling YaRN: What 128k Context Really CostsvLLM deleted the --rope-scaling flag in November 2025. The rope_parameters override that parses on v0.28.0, and what factor 4.0 charges every short prompt.
LLMS & MODELS06
Ollama num_ctx: Why Long Prompts Get Silently TruncatedOllama's default context window has been 262144, 32768 or 4096 since February 2026, picked from detected VRAM. Two truncation layers behave in opposite ways.
LLMS & MODELS07
llm-d vs Dynamo vs vLLM Production Stack: Which to Runllm-d v0.9.0 moved vLLM three minor versions in one release. Dynamo v1.4.1 moved none. The orchestration layer you pick is a change-control posture.
LLMS & MODELS08
Speculative Decoding Slower Than Baseline: How to Fix ItA 0.898 draft acceptance rate still lost to plain decode by 17 tok/s. The break-even is acceptance above draft cost, not a fixed percentage.
LLMS & MODELS09
AWQ vs GPTQ vs FP8: Which Quantization for Your GPULlama-3.1-8B is 16.06 GB in BF16 and 9.08 GB at FP8, not 8. The vLLM v0.27.1 support matrix, the Ampere fallback, and which format your batching wants.