Topic hub
LLMs & Models
Understand large language models, fine-tuning, prompt engineering, and model selection.
All posts · 53
LLMS & MODELS01
KV Cache Duplication Factor: Divide tp_size by KV HeadsYour KV cache is duplicated max(1, tp_size // H) times. At 8 kv-heads and -tp 8 that is 1. DeepSeek-R1's config.json says 128 and the real H is 1.
LLMS & MODELS02
vLLM rope_scaling YaRN: What 128k Context Really CostsvLLM deleted the --rope-scaling flag in November 2025. The rope_parameters override that parses on v0.28.0, and what factor 4.0 charges every short prompt.
LLMS & MODELS03
Ollama num_ctx: Why Long Prompts Get Silently TruncatedOllama's default context window has been 262144, 32768 or 4096 since February 2026, picked from detected VRAM. Two truncation layers behave in opposite ways.
LLMS & MODELS04
llm-d vs Dynamo vs vLLM Production Stack: Which to Runllm-d v0.9.0 moved vLLM three minor versions in one release. Dynamo v1.4.1 moved none. The orchestration layer you pick is a change-control posture.
LLMS & MODELS05
Speculative Decoding Slower Than Baseline: How to Fix ItA 0.898 draft acceptance rate still lost to plain decode by 17 tok/s. The break-even is acceptance above draft cost, not a fixed percentage.
LLMS & MODELS06
AWQ vs GPTQ vs FP8: Which Quantization for Your GPULlama-3.1-8B is 16.06 GB in BF16 and 9.08 GB at FP8, not 8. The vLLM v0.27.1 support matrix, the Ampere fallback, and which format your batching wants.
LLMS & MODELS07
Why Temperature 0 Gives Different Results on the Same Prompt1,000 runs of one prompt at temperature 0 returned 80 different completions. The cause is batch size, not sampling, and the fix lives in the server.
LLMS & MODELS08
Empty tool_calls With reasoning_content: vLLM Parser FixvLLM 0.19.0 returns 200 OK with a full reasoning field and tool_calls empty. Issue #39056 is still open. The real version floor is v0.24.0, not v0.20.0.
LLMS & MODELS09
Unsloth vs Axolotl vs LLaMA-Factory vs TRL: How to PickThe 3.2h vs 5.8h speed table everyone quotes has no date, no seed and no config. Primary-source VRAM and parallelism data pick the framework instead.