LLM Latency Targets: TTFT and Tokens/Sec SLOs in 2026

Methodology first, because a latency number without methodology is decoration.

The figures below come from the largest continuously running public LLM latency benchmark, snapshot taken 2026-07-27. Conditions: test client in Google Cloud us-central1-a, a roughly 10,000-token input workload (the benchmark default), fresh unique prompts per run to defeat provider-side caching, sampled about every three hours, reported as the median (P50) over the trailing 72 hours. These are P50 numbers, not P95, and they exclude every millisecond of cross-region round-trip between your users and the endpoint. Treat them as a floor, not a forecast.

One more caveat that most comparison tables get wrong: for a reasoning configuration, the reported first-token latency is effectively time to first answer token and it contains the entire thinking budget. Mixing those rows into the same table as non-reasoning configs is how you end up publishing "164 second TTFT" as if it were a prefill measurement. So they are split.

Two things to take from Table A. First, the fast tier is a narrow band: 0.75s to 1.6s to first token, from inside the datacenter, at 10k input. Second, output speed and first-token latency are almost uncorrelated. gpt-oss-120b is 7x faster than Claude Opus 4.7 on output speed and only 2.1x faster to first token, because TTFT is prefill plus queueing plus network while output speed is memory bandwidth. That split is the whole reason TTFT and inter-token latency are two separate budgets.

Serving path matters as much as model choice for the throughput half. Across specialist hosted inference providers measured by the same benchmark, median output speed ranges from about 35 tok/s to 578 tok/s on comparable open-weight models, while median first-chunk latency converges tightly at 0.7s to 1.0s across all of them. Buying a specialist serving stack buys output speed. It does not buy TTFT. For engine-level numbers on the same hardware, see our measured TTFT and inter-token latency by inference engine comparison.

If you set one latency SLO across an endpoint that serves both effort levels, that SLO is meaningless. The effort parameter alone spans 1.64s to 192.97s in this table.

These are our recommendations, derived from the measurements above plus human-perception research. They are not a published standard and no provider commits to them.

The grounding for the top three rows is human perception, not vendor marketing. The classic usability thresholds, established in 1993 and unchanged since because human attention has not been patched, are 0.1 second for a system to feel instantaneous, 1 second to keep the user's flow of thought uninterrupted, and 10 seconds to hold attention on the task at all. That is where 200ms, 800ms and 10s come from.

The voice rows come from turn-taking research rather than an engineering budget: a 2009 cross-linguistic study found conversational turn-transition offsets are unimodal with a modal gap between 0 and 200ms across typologically diverse languages. That is the real bar, and no LLM-in-the-loop pipeline currently meets it. We deliberately do not re-derive a voice number here; use the 300ms turn-latency budget for real-time voice with its stage-by-stage split.

Which exposes a hard architectural constraint worth stating plainly: a voice agent that needs 200 to 400ms of LLM TTFT cannot be running a 10,000-token prompt against a frontier reasoning model. Table A says the floor there is 750ms from inside the datacenter, before any network hop to your user. Meeting the voice budget means a small model, a short prompt, aggressive prefix reuse, or self-hosted inference next to the audio path. There is no fifth option.

Most latency arguments are definition mismatches. Microsoft's Azure OpenAI latency documentation gives the cleanest identity available from a first-party provider:

TTLT = TTFT + (TBT x Tokens Generated)

TTFT  Time to First Token: prompt submitted -> first token returned
TBT   Time Between Tokens: average gap between consecutive tokens
TTLT  Time to Last Token: total end-to-end response time

Two aggregation variants matter in practice. TPOT (time per output token) is (E2E latency - TTFT) / (output tokens - 1), computed per request and then averaged across requests. ITL (inter-token latency) measures the same gaps but aggregates token-weighted across all requests. When your request lengths vary, those two numbers diverge, and nearly every blog comparison silently mixes them.

The consequence Microsoft states outright is the one most teams get wrong: total latency scales with output tokens, so a rising end-to-end number is usually explained by longer outputs rather than a system regression. A 5-second response that produced 2,000 tokens and a 5-second response that produced 50 tokens are different events. Always pair a latency metric with a token-count metric or the alert is unactionable.

This also reconciles two heuristics that look contradictory. OpenAI's latency guidance says cutting 50% of output tokens may cut about 50% of latency, while cutting 50% of the prompt may only improve latency by 1 to 5%. Both are true: total request latency is decode-dominated, so prompt length barely moves it. But TTFT is entirely prefill, so prompt length moves TTFT substantially. If TTFT is your SLO, prompt length is your lever. If total time is your SLO, output length is.

Finally, the metric that should replace "average latency" on your dashboard: goodput, the percentage of requests meeting all SLOs simultaneously. Average latency hides the exact population you care about.

This is the part nobody answers honestly, so here it is flatly. Every major provider commits to availability. Exactly one publishes a numeric latency target, and it is expressed in tokens per second, not TTFT.

Azure OpenAI priority processing is the exception. Its documentation defines the target as "p50 request latency on a per 5 minute basis expressed as a percentile threshold" and publishes per-model values:

Three conditions attached to that commitment deserve to be in your risk register:

Elsewhere: OpenAI's direct priority processing documentation promises "significantly lower and more consistent latency" with no percentile, no window, no number and no service credits, enabled per request with service_tier: "priority". Anthropic's service tiers documentation describes standard tier as best-effort availability and targets 99.5% uptime on Priority Tier, which is an availability figure, not a latency one. As of this writing Anthropic's Priority Tier capacity commitments are no longer available for purchase, and Priority Tier is not supported on its newest flagship models at all. We found no published numeric latency commitment for Gemini either.

The practical read: if you need a TTFT guarantee, you are not buying it from a shared API. You are buying dedicated capacity or running the model yourself.

Look again at the effort ladders in Table B. GPT-5.6 Sol goes 2.97s, 4.48s, 11.02s, 129.47s across low, medium, high and max: a 44x swing. Claude Opus 5 goes 3.68s to 69.71s across low to max: 19x. Output speed over those same ladders moves from 70 to 76 tok/s and 54 to 53 tok/s respectively. The dial does not make the model generate faster or slower. It makes the model think longer before you see anything.

This broke the metric. Independent benchmarking now tracks two separate first-token metrics: time to first token (which for a reasoning model captures the first reasoning token) and time to first answer token, measured after thinking completes. If a vendor or a blog quotes a single TTFT for a reasoning model without saying which one, the number is unusable.

Two design consequences. First, effort level belongs in your SLO segmentation the same way prompt length does; one endpoint serving both low and high cannot have one target. Second, reasoning token counts are themselves highly variable between models of similar accuracy, which is its own budgeting problem covered in our analysis of reasoning-model token cost and overthinking.

Because only one of the four terms in TTFT is about the model. TTFT decomposes into queue wait, scheduler admission, prefill compute and network round-trip. Three of those are contention.

The dominant mechanism is prefill interference. When a long prompt is admitted into a running batch, its compute-heavy prefill stalls every decode already in flight, and users who were streaming smoothly hit a visible pause. That pause is your P99. Sarathi-Serve (arXiv:2403.02310, OSDI 2024) demonstrated the size of the effect by fixing only the scheduling: 2.6x higher serving capacity under tail-latency constraints on a 7B model with one A100, up to 3.7x on a 34B model with two A100s, and up to 5.6x on a 180B model with pipeline parallelism. Nothing about the models changed. The earlier chunked-prefill work behind it (arXiv:2308.16369) reported up to 10x decode throughput improvement from the same idea.

The hardware reason those two phases conflict is stark. A roofline analysis of transformer inference (arXiv:2402.16363) measured arithmetic intensity for Llama-2-7B on an A6000 at roughly 1024 to 1215 operations per byte during prefill projections and approximately 1 during decode. Three orders of magnitude apart, on the same weights.

The other tail inflators, in rough order of how often we see them:

Every number in Table A and Table B was measured from a test client inside us-central1-a. Your users are not there. Whatever your client-to-endpoint round-trip is, add it to every figure in this post, and add it again for each hop if you are proxying through a gateway.

Data residency changes the arithmetic in ways that rarely make it into latency planning. Azure's priority processing, the only published numeric latency target in the industry, is unavailable on EU data zone deployments. Anthropic applies a 1.1x capacity multiplier on both input and output tokens for US-only inference on its recent models. So a workload pinned to a jurisdiction is frequently a workload with fewer latency guarantees available, not more.

This is the honest case for on-premise and private deployment, and it is a narrow one. Self-hosting removes public-internet round-trip, cross-tenant queueing, provider-side ramp-rate downgrades, mandatory content-filter passes, and long-context tier downgrades. It removes none of the physics: prefill still costs compute, decode is still memory-bandwidth-bound, and your own scheduler now owns the tail behavior that Sarathi-Serve describes. In the regulated-industry deployments we work on at Particula Tech, the latency argument for on-premise is almost always about variance rather than mean: you trade a fast-but-uncontrollable shared endpoint for a slightly slower endpoint whose P99 you own. Measure your own baseline before assuming either direction.

Five rules, then the query.

One measurement confound worth knowing before you trust your own numbers: Microsoft documents that the requested generation size (max_tokens) is used as an initial estimate and compute time for the full size is reserved as the request is processed, so two requests generating the same number of tokens will differ in latency if their max_tokens differ. Pin it to a realistic ceiling per workload before you benchmark anything.

-- Goodput and tail latency, segmented by prompt-length bucket.
-- Run this instead of AVG(latency_ms).
SELECT
  CASE
    WHEN prompt_tokens <   2000 THEN 'a_under_2k'
    WHEN prompt_tokens <  10000 THEN 'b_2k_10k'
    WHEN prompt_tokens < 100000 THEN 'c_10k_100k'
    ELSE 'd_over_100k'
  END                                              AS prompt_bucket,
  COUNT(*)                                         AS requests,
  APPROX_QUANTILES(ttft_ms, 100)[OFFSET(50)]       AS ttft_p50_ms,
  APPROX_QUANTILES(ttft_ms, 100)[OFFSET(95)]       AS ttft_p95_ms,
  APPROX_QUANTILES(ttft_ms, 100)[OFFSET(99)]       AS ttft_p99_ms,
  ROUND(AVG(output_tokens / NULLIF(gen_seconds, 0)), 1) AS mean_tok_per_sec,
  ROUND(100 * COUNTIF(
      ttft_ms <= 800
      AND output_tokens / NULLIF(gen_seconds, 0) >= 30
  ) / COUNT(*), 2)                                 AS goodput_pct,
  ROUND(100 * COUNTIF(returned_tier != requested_tier) / COUNT(*), 2)
                                                   AS tier_downgrade_pct
FROM llm_requests
WHERE ts > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
  AND reasoning_effort = 'low'   -- segment by effort, never aggregate across it
GROUP BY prompt_bucket
ORDER BY prompt_bucket;

For load testing, mirror the benchmark methodology that produced Table A rather than inventing one: unique prompts per run so you are not measuring the provider's cache, fixed input-length buckets, a minimum output-token requirement, and enough repetitions across the day to catch diurnal load on shared endpoints. A single-shot latency check at 3am tells you nothing about your P95 at 2pm.

Then close the loop. A latency SLO that nobody alerts on is a document, not a control, and a green P95 dashboard is not the same thing as a healthy system: see instrumenting P95 latency alongside quality and cost for the monitoring layer that sits on top of these targets.

Set the targets from Table A and the workload table. Measure with the query. When you miss, the remediation options are in the fixes that actually move LLM latency once you miss the target, and if the answer turns out to be owning your own inference path, that is the work we do at Particula Tech.