Why Temperature 0 Gives Different Results on the Same Prompt

Two numbers from that experiment matter more than the headline.

The first is 78 out of 1,000 for the most common completion. This is not a distribution with one dominant answer and a rare glitch. The probability mass is spread across dozens of variants, so two randomly chosen long completions of the same prompt usually differ. If you have ever compared two production transcripts and assumed someone edited the prompt, this is the more likely explanation.

The second is token 103. Divergence does not start at the first token, because a short output has few opportunities to sit near a decision boundary. It accumulates. That single fact explains why almost nobody catches this before production: the standard smoke test is a handful of short completions on a development box, and a development box runs one request at a time.

At batch size 1, held constant, the reduction order never changes, so the server is reproducible. It stops being reproducible the moment other traffic shares the batch. This is the purest form of "works on my machine" available in AI infrastructure.

A related result sharpens where the risk sits. A paper submitted on 3 January 2026 looked at token probabilities rather than generated text and found the effect of GPU nondeterminism is significant for probabilities in the 0.1 to 0.9 band and much smaller when they approach 0 or 1. The practical reading: confident tokens are safe, contested tokens are where the flip happens, and you can estimate exposure from a single inference by inspecting token-level probabilities instead of running the same prompt a thousand times.

For a decision endpoint that emits a label, that is a cheap and specific control. Log the top-2 probability gap on the tokens that carry the decision. Anything in the contested band is a token whose value depends on batch composition.

Floating-point addition is not associative. Adding a large number to a small one loses low-order bits, so (a + b) + c and a + (b + c) can differ in the last place. High-performance GPU kernels exploit that freedom aggressively: they split a reduction across a number of parallel units chosen from the total amount of work in flight, then combine partial sums in whatever order finishes. Change the work, change the split, change the order, change the last bits.

Dynamic batching changes the work on every iteration. Continuous batching admits and retires sequences mid-flight, so between step 40 and step 41 your sequence can go from sharing the batch with 3 others to sharing it with 47. Nothing about your request changed. The arithmetic did.

Exactly three operations carried the variation end to end, and all three had to be made batch-invariant before the thousand completions collapsed into one: RMSNorm, matrix multiplication, and attention. Attention is the hardest of the three: the reduction runs over the key and value sequence, and the split between cached and freshly processed tokens shifts with chunked prefill, so the reduction strategy has to be fixed independently of how the sequence is chunked.

Two implications fall out immediately.

Nondeterminism is not a model property. The same weights on a serialized single-stream server are reproducible. Swap in dynamic batching and they are not, so no amount of model-level testing tells you anything about it.

And it is not an atomics problem, which is the most common wrong diagnosis. The standard forward pass does not need atomic accumulation to be nondeterministic. Batch-size-dependent reduction order is enough on its own.

Both major open-source engines now ship this behind a flag.

# vLLM: batch-invariant kernels. Must be set before the process starts.
VLLM_BATCH_INVARIANT=1 vllm serve Qwen/Qwen3-8B

Requirements are documented: NVIDIA GPUs at compute capability 8.0 or higher, and the flag disables optimizations that would reintroduce nondeterminism, including custom all-reduce in tensor-parallel mode. It is still marked beta, with DeepSeek, Qwen3, Qwen2.5, Llama 3, GPT-OSS, Mistral and Phi among the tested families.

# SGLang: deterministic serving path
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --attention-backend fa3 \
    --enable-deterministic-inference

SGLang restricts deterministic inference to three attention backends, and they are not interchangeable.

FlashAttention 3 is the default and the only backend that keeps prefix caching, which matters if your deterministic path serves anything with a shared system prompt. Losing the radix cache on FlashInfer is a second throughput hit stacked on the first, and prefix reuse is the whole reason SGLang wins the workloads it wins.

Determinism above temperature 0 is supported through an explicit sampling_seed, which defaults to 42. Different seeds give you diverse but reproducible draws, which is what you want for anything that samples multiple candidates.

"sampling_params": {"temperature": 0.8, "sampling_seed": 42}

The bill is documented on both sides. SGLang reports most slowdowns between 25 and 45 percent, with a 34.35 percent average across the FlashInfer and FlashAttention 3 backends, and attributes most of it to unoptimized batch-invariant matrix multiplication and attention rather than to anything fundamental. The September 2025 measurement agrees in wall clock.

The gap between 55 and 42 seconds is the part being actively optimized. Plan against a third less throughput today, and do not build a capacity model that assumes the improvement.

The kernel approach has a structural cost: determinism is welded to kernel design, so every new kernel needs a batch-invariant twin, and you pay the fixed overhead on all traffic whether or not that traffic will ever be replayed.

LLM-42, submitted 25 January 2026, takes the other route. It borrows the shape of speculative decoding: decode tokens on the normal nondeterministic fast path, then run a lightweight verify-rollback loop where the verifier replays candidate tokens under a fixed-shape reduction schedule, commits the ones guaranteed to be consistent across runs, and rolls back the ones that violate determinism. It works because a sequence in a consistent state usually emits a consistent next token even under dynamic batching, and most kernels already use shape-consistent reductions. Existing kernels stay largely unchanged, and overhead scales with the share of traffic that requires determinism.

That last property is the interesting one for a mixed workload. If 4 percent of your requests are decisions that must be replayable and 96 percent are drafting and summarization, a flat 34 percent tax on everything is the wrong shape of bill.

There is a training-side dividend worth naming. When the sampler and the trainer disagree numerically, a nominally on-policy reinforcement learning run is quietly off-policy, and the measured KL divergence between sampler and trainer sits around 0.001 even with off-policy correction applied. With deterministic inference it goes flat to 0, which makes the run genuinely on-policy and removes the correction term. If you are doing RL fine-tuning inside your own perimeter, that is a correctness argument, not a convenience one.

This is the part that changes what you can honestly write in a control document.

Everything that decides whether two runs agree lives on the server: batch composition and admission policy, the kernel build, the attention backend, tensor-parallel layout, GPU generation, and whether prefix cache hits changed how a prefill was chunked. A client sends temperature and seed. Neither reaches any of that.

For an API consumer, the consequences are concrete. You cannot enable batch-invariant kernels on someone else's fleet. You cannot pin the GPU generation your request lands on. You are not told the batch you shared. And the model version that made a decision can be deprecated long before anyone asks you to reproduce it, which turns a replay requirement into a hardware and licensing problem you have no standing to solve. The defensible control on a hosted endpoint is retention: the exact request, the exact response, the version string, hashed and stored. Not replay.

On-premise is where the answer actually changes, and not for the usual data-residency reason alone. You control admission, so you can hold a deterministic path at a fixed configuration. You can pin the container digest rather than a version string, which is what pins kernels. Weights stay on your storage, so a two-year-old decision is still reproducible because nobody deprecated the model out from under you, the same argument that makes air-gapped vLLM deployment worth the setup cost.

There is a data-protection point that is easy to miss. Replay means sending the original prompt through the model again. If that prompt contains patient records, account details or case files, replaying it on a hosted API is a second disclosure of the same data to the same processor, and it belongs in your processing agreement and your retention schedule. Inside the perimeter, replay is an internal operation and raises no new disclosure at all. For anyone working through AI compliance in financial services, that distinction is usually worth more than the throughput conversation.

Do not make production pay 34 percent for a property that only the decision-bearing slice of requests needs. Split the fleet.

The production path runs default kernels and dynamic batching, tuned for throughput and tail latency exactly as it is today, including prefill-decode disaggregation if you are at that scale. It is nondeterministic and you say so in writing, with a measured divergence rate attached.

The replay path runs batch-invariant kernels at a pinned configuration, sized small because it only serves replays, evals and any request class flagged as decision-bearing at admission. It is also where evaluation suites belong: regression scores that move because the batch moved are noise you will spend weeks chasing. That failure mode is why evals-driven development needs a fixed serving configuration under it, and why an unstable serving path makes LLM-as-judge regression testing even harder to trust than it already is.

What makes replay possible is not the second endpoint. It is that every decision carries the fingerprint of what produced it.

decision_id: 8f42c1a0-...
model:
  name: qwen3-8b-instruct
  weights_sha256: 3b1f9c...        # the weights, not the alias
serving:
  engine: sglang
  image_digest: sha256:5c1d0e...   # pins kernels; a version string does not
  attention_backend: fa3
  deterministic: true              # --enable-deterministic-inference
  tensor_parallel_size: 2
  accelerator: h100-80gb
request:
  temperature: 0
  sampling_seed: 42
  max_new_tokens: 512
  prompt_sha256: 7d90ab...
output:
  text_sha256: c41a77...
  decision_token_top2_gap: 0.41    # contested band, flagged for review

Three fields do the heavy lifting. The image digest pins kernels, which a semantic version does not. The weights hash pins the model against a moved alias. The top-2 gap tells a reviewer whether this specific decision sat near a boundary where reduction order could have flipped it.

If replaying a decision reproduces the recorded output hash on the deterministic path, you have evidence. If it does not, you have a config drift incident with a precise diff, which is a far better position than a shrug.

Controls here are usually written to describe an outcome rather than a mechanism, so the literal text asks for more than the intent requires. Read the intent before you buy the throughput.

The language we would put in the control is deliberately narrow: decisions of a defined class are served on a configuration pinned by weights hash and image digest with batch-invariant kernels enabled, each decision record carries that fingerprint, and any decision can be re-derived on the replay path within a stated window. That is auditable, testable, and cheap, because it applies to the decision class rather than to all traffic.

The claim to avoid is that all model output is bit-identical. It is not, it costs a third of your throughput to make it so, and the first person to run two production requests through a nondeterministic path will disprove it. This is the same discipline required for testing systems that have no single right answer: state the property you actually hold, measure it, and do not promise the stronger one. More on serving and model selection sits in the LLM models pillar.

You need one number before any of this becomes a plan: your divergence rate at production concurrency. It takes an afternoon.

Put the prompt in payload.json at temperature 0 with a realistic max_tokens, at least 512, because divergence accumulates and short outputs hide it.

probe () {   # $1 = concurrency
  seq 200 | xargs -P "$1" -I{} \
    curl -s http://localhost:30000/v1/completions \
      -H 'content-type: application/json' -d @payload.json \
  | jq -r '.choices[0].text | @base64' \
  | sort | uniq -c | sort -rn
}

probe 1    # quiet server, near constant batch
probe 64   # saturating concurrency, batch shape moves every step

Report two lines: unique completions at concurrency 1, and unique completions at saturating concurrency. If the first is 1 and the second is not, you have reproduced the problem on your own stack with your own model, and you have the number that decides whether this is a footnote or a project.

Then run the same probe against a server started with --enable-deterministic-inference or VLLM_BATCH_INVARIANT=1 and record the wall clock difference. Now you have both sides of the trade, measured on your hardware rather than borrowed from someone else's benchmark, and the conversation with model risk becomes an arithmetic problem instead of an argument.

One thing to fix immediately either way: if any request in your system carries a decision, start logging the serving fingerprint next to it today. Enabling deterministic kernels later is a flag. Reconstructing what configuration produced last quarter's decisions is not.