Ollama num_ctx: Why Long Prompts Get Silently Truncated

At startup the server sums total VRAM across detected devices as totalVRAM += gpu.TotalMemory - envconfig.GpuOverhead(), then runs a three-way switch.

The thresholds are 47 and 23 rather than 48 and 24 on purpose: the comment calls them "slightly lower thresholds (47/23 GiB vs. 48/24 GiB) to account for small differences in the exact value". OLLAMA_GPU_OVERHEAD is subtracted per device before the sum, so raising it can drop a host a tier, and two 24 GB cards reach the top tier where one reaches the middle. Which card you actually bought decides the tier before any config file does.

The decision is announced once at slog.Info, and the binary's help text documents the tiering the FAQ page does not.

# The tier decision is logged once, at INFO, when the server starts
journalctl -u ollama --no-pager | grep 'vram-based default context'
# level=INFO msg="vram-based default context" total_vram="23.6 GiB" default_num_ctx=32768

# And the binary's own help text names the tiers
ollama serve --help 2>&1 | grep OLLAMA_CONTEXT_LENGTH
# OLLAMA_CONTEXT_LENGTH   Context length to use unless otherwise specified (default: 4k/32k/256k based on VRAM)

Old advice is stale twice over: commit 424f6486 of 2025-04-22 raised the unspecified-context fallback from 2048 to 4096, and February 2026 replaced that with the tiering.

DefaultOptions() in api/types.go sets NumCtx: int(envconfig.ContextLength()), so with the environment variable unset the request arrives carrying NumCtx: 0 and server/routes.go substitutes opts.NumCtx = s.defaultNumCtx. Precedence lives in usesAutomaticNumCtx: false if requestOpts["num_ctx"] is present, false if model.Options["num_ctx"] is present, otherwise envconfig.ContextLength() == 0.

Four surfaces set the window; the table adds a fifth row for when none of them do.

On the native API the field sits inside options, never at the top level. And what you ask for is not always what you get: getRunner floors NumCtx at 4, and at 2048 for any model with the vision capability, while effectiveContext caps it at the GGUF's trained context and optionsForPrompt clamps it again to the loaded runner's context. Asking for more than the runner has is silently reduced, not refused.

Ollama says this itself. Under "Setting the context size", its OpenAI compatibility page states: "The OpenAI API does not have a way of setting the context size for a model."

"Unsupported" and "rejected" are not the same failure. server/routes.go registers /v1/chat/completions as middleware.ChatMiddleware(), s.ChatHandler and /v1/completions as middleware.CompletionsMiddleware(), s.GenerateHandler: the OpenAI routes are the native handlers behind a translating middleware. That middleware decodes into ChatCompletionRequest, which carries eighteen fields and no NumCtx, Truncate or Shift, so an unmapped JSON key is discarded during decoding. FromChatRequest then builds the options map with at most seven keys: stop, num_predict (from max_tokens), temperature, seed, frequency_penalty, presence_penalty and top_p.

Because the middleware never puts num_ctx into requestOpts, usesAutomaticNumCtx returns true for every /v1 request unless a Modelfile or the server environment variable set it. Your context setting is not overridden. It is never seen.

# num_ctx is not a field on ChatCompletionRequest: the key is dropped, response 200
curl -s http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"llama3.2","messages":[{"role":"user","content":"hi"}],"num_ctx":32768}'

# Reachable from /v1 instead: bake it into the model
printf 'FROM llama3.2\nPARAMETER num_ctx 32768\n' > Modelfile
ollama create llama3.2-32k -f Modelfile
# then call /v1 with "model": "llama3.2-32k"

Do not reach for a query parameter, a header or a vendor extension object. None exists: a request to add num_ctx as an extra query parameter was filed in October 2024 and is still open, and a pull request titled "openai: forward num_ctx to options in OpenAI-compatible endpoints" was opened on 2026-06-20 and remains unmerged.

This refines our Ollama versus vLLM comparison: for chat, drop-in is a fair claim, but for context control it is not, because num_ctx, truncate and shift have no OpenAI-side equivalent. Whether that argues for moving the workload off Ollama entirely is decided elsewhere. The same problem appears one layer up in the chat UI, where a token limit set in the interface trims before the request is sent.

server/prompt.go states its contract in a doc comment: "chatPrompt truncates any messages that exceed the context window of the model, making sure to always include 1) the latest message and 2) system messages".

The implementation walks an index forward from the start of the conversation, collecting every message in the span it is about to discard whose role is system and re-prepending it to the surviving slice, while a guard, if i == lastMsgIdx { currMsgIdx = lastMsgIdx; break }, protects the final message. Whole turns disappear from the front, the system prompt survives the cut, and the newest user message is never the casualty.

Two details change how you test it. Images count 768 tokens each toward the budget when the model has projector paths (imageNumTokens := 768). And truncateNativeChatMessages opens with if !truncate || opts == nil || opts.NumCtx <= 0 || len(req.Messages) <= 1 { return req.Messages, nil }: one message returns immediately, so a single giant prompt never exercises message-level truncation, and stuffing a canary into the head of one enormous request proves nothing here.

The only announcement is at debug level, in two lines that disagree. server/prompt.go logs truncated as len(msgs[currMsgIdx:]), the number of messages kept; server/routes.go logs the same truncated key as currMsgIdx, the number dropped. Since LogLevel() returns slog.LevelInfo unless OLLAMA_DEBUG is set, neither line appears on a stock server.

If the rendered prompt still does not fit, the runner takes over, with rules nothing like layer one's. llm/llama_server.go sets fullPromptLimit := s.options.NumCtx - 1. With context shift disabled the server returns HTTP 400; with it enabled, the default for everything except models whose family contains deepseek2, it computes:

nKeep := req.Options.NumKeep          // 4 by default
if s.tokenizerAddsBOS() { nKeep++ }   // 5 with a BOS-adding tokenizer
limit := contextShiftPromptLimit(s.options.NumCtx, nKeep)
discard := len(tokens) - limit
truncated = tokens[:nKeep] + tokens[nKeep+discard:]

NumKeep is 4 in DefaultOptions(), with the comment "set a minimal num_keep to avoid issues on context shifts". That is four tokens, not four messages or turns. Whatever your system prompt says, at this layer it is gone.

The limit is the counterintuitive part. contextShiftPromptLimit(numCtx, numKeep) returns numCtx - max((numCtx-numKeep)/2, 1), described in the source as matching "the old runners' first context shift: preserve num_keep, then free roughly half of the remaining context". At numCtx 4096 with nKeep 5 that is 4096 - max((4096-5)/2, 1) = 4096 - 2045 = 2051, or 50.1 percent of the window. At 32768 it is 16387; at 262144 it is 131075. The first shift frees half the window, not the overflow.

Work a case. A 12,000-token rendered prompt into a 4096-token window with nKeep 5 gives a limit of 2051 and a discard of 9,949, so the surviving prompt is tokens[0:5] plus tokens[9954:12000]: 5 + 2,046 = 2,051 tokens. The model sees your first five tokens and your last 2,046.

Unlike layer one, this one talks: slog.Warn("truncating input prompt", "limit", limit, "prompt", len(tokens), "keep", nKeep, "new", len(truncated)), emitted at the default INFO level, so the line is already in a stock log.

So "Ollama drops the head and keeps the tail" fits both layers; only one of them spares the system prompt.

api/types.go declares Truncate *bool and Shift *bool on both GenerateRequest and ChatRequest, described in the struct as truncating "the chat history messages if the rendered prompt exceeds the context length limit" and shifting "the chat history when hitting the context length limit instead of erroring". Both default to true when omitted (req.Truncate == nil || *req.Truncate). Setting shift: false converts a silent trim into a failure you can alert on.

curl -s http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "..."}],
  "options": { "num_ctx": 32768 },
  "shift": false,
  "stream": false
}'

With shift off and a prompt longer than NumCtx - 1, the runner returns a 400 instead of trimming:

{
  "error": "the prompt is longer than the context length currently available to the model; shorten the prompt, adjust the context length in settings, or use a model with a longer context length"
}

Neither field exists on any /v1 schema, and FromChatRequest never assigns them, so both arrive nil and default to true for every OpenAI-compatible request: on /v1 you cannot ask Ollama to fail loudly.

Embeddings have their own truncate, and it is opt-in the same way. When req.Truncate is non-nil and false and the token count exceeds the context, the handler returns a 400 carrying "the input length exceeds the context length", and normalizeEmbeddingError maps five runner-error substrings onto that message. Leave truncate out and the same input is cut and returns 200 with an embedding of the first part of the document.

curl -s http://localhost:11434/api/embed -d '{
  "model": "<your-embedding-model>",
  "input": "<input longer than the model context>",
  "truncate": false
}'
# {"error":"the input length exceeds the context length"}

Set truncate: false on every embedding call in an ingestion pipeline: a silently truncated embedding is a corrupted index entry that never announces itself.

The window you set is not the window that gets allocated. server/sched.go computes effectiveLlamaServerContext as the effective per-model context multiplied by max(numParallel, 1), and Ollama's FAQ says the same in prose: "a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation". OLLAMA_NUM_PARALLEL defaults to 1, so this bites only once you raise it, multiplicatively, against the tier the host landed in; the per-token KV cache arithmetic from the vLLM side applies once you multiply by the slot count. Embedding models are forced to one slot, as are eleven named architecture families including mllama, qwen3vl and the nemotron_h variants.

The sharper failure on a shared host is reload thrash. needsReload compares the loaded runner's Options.Runner against the request's with reflect.DeepEqual, and NumCtx is part of Runner. The escape hatch is narrow: if runner.numCtxAuto && req.numCtxAuto { optsNew.NumCtx = optsExisting.NumCtx }. Two callers that both leave num_ctx unset share a runner; one caller setting it to a different value breaks the equality and starts an unload-reload cycle on every alternation, felt by other tenants as latency.

Pinning costs one more thing. When num_ctx is automatic and a load fails on out-of-memory, reduceAutoNumCtxForLoadOOM retries once at the next lower tier via nextLowerAutoNumCtx, stepping 262144 to 32768 to 4096. Set num_ctx explicitly and that function returns immediately: the load fails.

This is where the tiering stops being trivia. A private deployment is the case where one model image lands on hardware that differs by site: a 24 GB workstation card in a branch office, a 48 GB card in the datacentre, a CPU-only box in an air-gapped environment. Identical containers, Modelfiles and client code then produce a 4096-token window at one facility and 262144 at another, announced only in a single startup line. It presents as "the model is worse at the branch", and the failing artefact is a shortened prompt, not an error.

Three things follow. Pin the window in configuration management, not in inference: set OLLAMA_CONTEXT_LENGTH in the unit file, and size the pin against the smallest card in the estate, because pinning forfeits the step-down above. While you are in that unit file, hardening the Ollama server itself covers the binding and authentication side. Treat truncation as a records-integrity event, not a performance one. If a control requires you to show a document was fully considered, a layer-two shift keeping five head tokens and half a window of tail breaks that claim, and nothing in the response records it, so the audit trail must come from the truncating input prompt log line shipped with the request record. Do not give one host two windows. Runners are keyed by the model's blob path, so a second model name over the same weights shares the runner, and a Modelfile PARAMETER num_ctx turns usesAutomaticNumCtx false exactly as a request option does.

Start with what the runner reports:

ollama ps
# NAME    ID    SIZE    PROCESSOR    CONTEXT    UNTIL

curl -s http://localhost:11434/api/ps | jq '.models[] | {name, context_length, size_vram}'

Then prove which turns survive, without spending a generated token. _debug_render_only is a real request field on both the native and the OpenAI routes, and the response carries _debug_info.rendered_template. That render happens after layer one and before layer two, so what comes back is the prompt layer two would then receive. The probe has to be a conversation: truncateNativeChatMessages short-circuits on len(req.Messages) <= 1, so one oversized message renders back identical to its input.

{
  "model": "llama3.2",
  "_debug_render_only": true,
  "messages": [
    {"role": "system", "content": "SENTINEL-SYSTEM. Answer only with the sentinel you were given."},
    {"role": "user", "content": "SENTINEL-001 <thousands of tokens of filler>"},
    {"role": "assistant", "content": "ack 001"},
    {"role": "user", "content": "SENTINEL-002 <thousands more>"},
    {"role": "assistant", "content": "ack 002"},
    {"role": "user", "content": "Which sentinels are still in your context?"}
  ]
}

curl -s http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d @probe.json | jq -r '._debug_info.rendered_template' | head -c 400

Missing SENTINEL turns mean layer one fired; every turn intact rules it out. The debug render returns before the model runs, so layer two's warning comes from sending the same conversation for real. To see layer one at all, raise the log level:

# Layer one announces itself at DEBUG only, and the server runs at INFO by default
OLLAMA_DEBUG=1 ollama serve
# level=DEBUG msg="truncating native chat messages which exceed context length" truncated=4

# Layer two announces itself at WARN, so it is already in a stock log
journalctl -u ollama --no-pager | grep 'truncating input prompt'
# level=WARN msg="truncating input prompt" limit=2051 prompt=12000 keep=5 new=2051

OLLAMA_DEBUG takes 0 or false for INFO (the default), 1 for DEBUG and 2 for TRACE. OLLAMA_DEBUG_LOG_REQUESTS logs inference request bodies and replay curl commands to a temporary directory: fine on a bench box, inappropriate on anything holding regulated data.

Three of those signals are visible on a stock server: vram-based default context at startup, truncating input prompt per request, context_length on /api/ps. One needs OLLAMA_DEBUG=1, one needs _debug_render_only. The chat response body carries none of them, on any endpoint, which is what is genuinely silent to the caller.

Do one thing this week: run ollama ps on every inference host you operate and write the CONTEXT column next to the card in each box. If two hosts running the same model report different numbers, you have found the split before a user does, and the fix is one line in a unit file. The rest of the local-serving decisions sit in our LLMs and models pillar.