Text Embeddings Inference vs vLLM: Pick an Embedding Server

The flags show the shape of the feature. --runner accepts auto, draft, generate and pooling, defaults to auto, and the docs note that each vLLM instance supports one model runner even when the same model could serve multiple types. --convert accepts auto, classify, embed and none, also defaulting to auto, and its documented use case is adapting a generation model for pooling tasks. --pooler-config takes a valid JSON string or JSON keys passed individually.

That is a well-built adapter layer on a generation engine, and the docs decline to claim it is fast. Serve embeddings from vLLM when the win you want is one fewer piece of software to run and approve, and run a dedicated embedder otherwise.

The API surface is generous either way. Offline: LLM.embed, LLM.classify, LLM.score and the generic LLM.encode with a pooling_task argument. Online: an OpenAI-compatible /v1/embeddings alongside /classify, /v1/score and /pooling. Four pooling task categories are documented: classify and embed are sequence-wise, token_classify and token_embed token-wise.

TEI is Apache-2.0, written in Rust, created in October 2023, and sits at roughly 5,000 GitHub stars with about 430 forks. Its real constraint is not speed, it is the architecture list. TEI supports "Nomic, BERT, CamemBERT, XLM-RoBERTa models with absolute positions, JinaBERT model with Alibi positions and Mistral, Alibaba GTE, Qwen2 models with Rope positions, MPNet, ModernBERT, Qwen3, and Gemma3", and for sequence classification only CamemBERT and XLM-RoBERTa. The set is enumerated rather than open-ended, so an unsupported architecture will not load, and that fact decides TEI versus Infinity before any benchmark is read. Note also that the documented --dtype values are float16 and float32: no int8, no fp8.

Infinity is MIT, created two days earlier in October 2023, at roughly 2,900 stars. Its description reads "a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali", and the README adds ColBERT late-interaction embeddings and text classification. Its --engine flag picks between three backends with real constraints: torch requires a model compatible with sentence-transformers and AutoModel, optimum requires an ONNX file, and ctranslate2 supports BERT models exclusively. Acceleration is documented for NVIDIA CUDA, AMD ROCm, CPU-only, AWS INF2 and Apple MPS, a longer hardware list than TEI documents.

vLLM's pooling runner is the adapter described above. The documented offline sample:

from vllm import LLM

llm = LLM(model="intfloat/e5-small", runner="pooling")
(output,) = llm.encode("Hello, my name is", pooling_task="embed")
data = output.outputs.data

Serving a generation checkpoint as an embedder is the --convert case, and the pooling method goes in the JSON config rather than in a dot-notation flag:

vllm serve $model \
  --runner pooling \
  --convert embed \
  --pooler-config '{"pooling_type": "MEAN"}'

The keys you may put in there are the fields of vLLM's PoolerConfig: task, pooling_type, seq_pooling_type, tok_pooling_type, use_activation, dimensions, enable_chunked_processing, max_embed_len, logit_mean, logit_sigma, step_tag_id and returned_token_ids. Sequence pooling accepts CLS, LAST and MEAN; token pooling accepts ALL and STEP. There is no normalize key and no softmax key.

Arctic Inference is a separate artifact from the third, not a configuration of it. Apache-2.0, created in March 2025, at roughly 470 stars, and installed rather than configured:

pip install arctic-inference[vllm]
export ARCTIC_INFERENCE_ENABLED=1
vllm serve $model --runner pooling

In a Python script the equivalent is calling vllm.plugins.load_general_plugins(). You cannot reach its numbers by tuning stock vLLM flags, because they come from code the plugin adds.

Start from the baseline the TEI README gives you, then swap the tag for the one matching your compute capability. The README lists no latest:

docker run --gpus all -p 8080:80 -v $volume:/data --pull always \
  ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 \
  --model-id $model

Eight tags and one gap. Volta is listed as not supported, and the Turing and both Blackwell images carry an experimental marker, so a B200 node is not a copy-paste deployment. The container publishes port 80, which the run command maps to 8080; the CLI's own --port default is 3000, so do not assume the two agree.

Now the arithmetic. --max-batch-tokens defaults to 16384, and the README calls it one critical control for maximum usage of the available hardware: it is the total pool of tokens the batcher will put in one forward pass. --max-client-batch-size defaults to 32 and controls how many inputs one client request may carry. TEI's own README benchmark runs at a sequence length of 512 tokens, and 32 times 512 is exactly 16,384. At that length, one maximal client request fills the token budget precisely.

Move off 512 and the two numbers stop lining up. At 400-token chunks, a maximal request carries 32 times 400, which is 12,800 tokens, or 78 percent of the budget. A single-threaded ingest client cannot fill a batch, so you need at least two requests in flight before the batcher does what the flag promises. That concurrency figure comes from two documented defaults, not from a chart.

Two rows in that table bite on ingest. --payload-limit is a second, independent ceiling: 2 MB of request body regardless of how many inputs --max-client-batch-size permits, so long chunks hit the byte limit before the count limit. And --tokenization-workers resolves at startup to the CPU core count minus one, clamped between 1 and 64, counted against the cores the container is actually given rather than the cores in the host. Run TEI with a 2-core quota next to an A10 and one tokenizer worker feeds the GPU.

A token budget is a memory bound, not a work bound. Inputs of mixed length get padded to the longest sequence in the batch, so a batch holding one 500-token chunk and thirty-one 90-token chunks does roughly 32 times 500 tokens of work while the budget accounting says it did far less. Nothing in either server's defaults fixes this for you.

Both projects have an open issue about it. TEI issue #65, "Latency-Throughput trade-off: Queue ordering to minimize padding", was opened in November 2023 and is still open with no comments. Its proposal: when the queue is deeper than the maximum batch size, pop a multiple of that size (the worked example is 8 times 32), sort the popped items by length, and form 8 length-homogeneous batches, cutting padding without changing the batch size. The issue notes there is no effect when the queue sits below the server's maximum batch size, which is the honest caveat: it only helps once you are saturating the server.

It was filed by the author of the Infinity server, who implemented the sort in that project and noted that sentence-transformers does the same. That provenance makes it a shared, unsolved surface in embedding serving rather than a talking point one project uses against another.

The mirror image sits on Infinity. Issue #87, "Adding max token budget per batch", was opened in February 2024 and is also still open with no comments: batching by count alone can blow up memory, the example being a batch of 64 against an 8k-context BERT at 64 by 8192, and it proposes a token budget in the style of 64 by 512. TEI batches by token budget and lacks length sorting; Infinity sorts by length and batches by count. Each open issue asks for the thing the other project already has.

Two cautions on that history. The 64 in issue #87 was the default as it stood in February 2024; the currently documented v2 CLI default is --batch-size 32. And a separate Infinity issue from October 2023, filed by an outside contributor, argued that random batch scheduling could starve individual requests at very deep queues. It was closed six days later, so it is history, not a live bug.

TEI v1.9.0 was published on 17 February 2026 with two breaking changes. The first is that --auto-truncate now defaults to true, meaning sequences are truncated to the lower value between --max-batch-tokens and the maximum model length. The second is that the default GeLU implementation became the tanh approximation instead of exact GeLU, which is a useful reminder that a point release can move your numerics under you.

The first changes a failure mode rather than a failure rate. Before v1.9.0 an over-length input was rejected and your ingest job saw an error. After v1.9.0 the same input returns a correctly shaped vector computed from only the tokens that fit: your index accepts it, retrieval quality drops, and the cause is a server default you never set.

Two rows there carry gaps rather than answers, and they stay gaps. Infinity's behaviour on an over-length input is not stated in its README or its v2 CLI reference, and vLLM's request-level truncation behaviour on /v1/embeddings was not confirmed. Filling them by inference is how wrong tables get written.

For an ingest run, turn truncation off:

docker run --gpus all -p 8080:80 -v $volume:/data \
  ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 \
  --model-id $model \
  --max-batch-tokens 16384 \
  --max-client-batch-size 32 \
  --max-concurrent-requests 512 \
  --auto-truncate false

Every value except the last is the documented default, written out so the batching contract is visible in the run command. --auto-truncate accepts an explicit value because its definition takes an optional argument, so AUTO_TRUNCATE=false is equivalent. A 512-token model handed a 900-token chunk then gives you this instead of a vector:

HTTP 422 Unprocessable Entity
{"error":"Input validation error: `inputs` must have less than 512 tokens. Given: 900","error_type":"Validation"}

The check in TEI's tokenization path is a strict greater-than against the maximum input length, and the router maps validation errors to 422. This path is not a 413.

Better than catching 422s is not needing to. TEI's /tokenize route runs the server's own tokenizer, so you can measure real token lengths against the model cap before the ingest starts:

curl -s http://localhost:8080/tokenize \
  -H 'Content-Type: application/json' \
  -d '{"inputs": "...one chunk of your corpus..."}'

Send a sample of chunks through that route, count the tokens each returns, and compare against the input-token cap for your model. If the tail of your length distribution sits above the cap, the fix belongs upstream in how you size chunks, not in a server flag.

Sizing an ingest run is one division. Two million chunks at 400 tokens each is 800,000,000 tokens. At a sustained 25,000 tokens per second, that is 800,000,000 divided by 25,000, or 32,000 seconds: 8.9 hours.

The divisor is the part no published chart can give you. 25,000 tokens per second is a placeholder in that worksheet, not an expectation. Measure it by pushing a few thousand real chunks at your real chunk length through the server you picked, on the card you own, and reading the rate off /metrics. Published charts fix a sequence length and a GPU; your corpus has a length distribution, and padding waste means the mean chunk length is not what sets your rate.

The concurrency you need to reach that rate falls out of the same two defaults. At 512-token chunks a single sequential client already presents full batches. At 150-token chunks one maximal request carries 4,800 tokens, under a third of the budget, so you need four or more concurrent clients before the batcher sees a full batch. Whatever number you land on has to stay under --max-concurrent-requests, which defaults to 512 and sheds load above it.

That rate also decides whether an index-time enrichment step is affordable, since rewriting every chunk with an LLM before embedding multiplies the run by a factor you can only compute once you know it. That is the arithmetic behind the index-time cost of contextual retrieval.

The throughput figures in circulation for embedding serving trace back to one published benchmark. Snowflake published an engineering blog post on 29 May 2025 reporting 16x and 4.2x higher embedding throughput for short sequences of 50 tokens and long sequences of 512 tokens compared to vLLM, and separately 2.4x higher embedding throughput than Text Embeddings Inference for short sequences with performance parity for longer sequences. The benchmark ran on an H200 GPU using snowflake-arctic-embed-m-v1.5, Snowflake's own embedding model. The post notes that Snowflake itself served embeddings in production on A10G at the time and extended the evaluation to the H200 as more powerful hardware.

Three things sit behind those numbers.

The subject is Arctic Inference, not vLLM. Those gains belong to an open-source vLLM plugin Snowflake released, and you get them by installing a package and setting an environment variable, not by tuning flags. The post names three optimizations behind them: encoding the embedding vector as little-endian bytes, disaggregating tokenization from inference, and running multiple identical models on one GPU. It does not attribute a share of the speedup to any one of them.

Read the direction of the 2.4x carefully. It is Arctic Inference ahead of TEI on short sequences with parity on long ones, not TEI ahead of vLLM.

And the provenance is vendor, model, hardware and date all at once: a vendor benchmarking its own plugin against its own model on an H200, which the same post describes as supply-constrained, roughly fifteen months before this post. The ArcticInference README's own headline figure, 1.4M tokens per second per GPU for embeddings with the plugin plus vLLM, is a project self-report and should be read as one.

No independent third-party head-to-head of TEI, Infinity and vLLM pooling on identical hardware with an identical model was located for this comparison. Every published throughput number located for it comes from a party with a stake in the outcome, which is why this post ranks nothing on speed and argues from defaults, failure modes and process count instead.

Inside a regulated perimeter the embedding server stops being a throughput question and becomes a process-count question. Every long-lived process on a GPU node is another artifact to patch, another port to justify, another line in the change record. Three facts do the work.

TEI serves one model per process. An embedder plus a reranker is therefore two containers, two ports and two sets of weights resident on the card. Infinity's documented pattern is to repeat --model-id and get both in one process on one port:

infinity_emb v2 \
  --model-id BAAI/bge-small-en-v1.5 \
  --model-id mixedbread-ai/mxbai-rerank-xsmall-v1 \
  --batch-size 8 \
  --port 7997 \
  --device cuda \
  --dtype auto \
  --engine torch

The --batch-size 8 is deliberate: the documented v2 CLI default is 32, and the README's multi-model rules say a single value is broadcast to every --model-id in the invocation, so both models get 8. The lower number is the visible cost of two models sharing one card. Which reranker belongs in that second slot is a different question, answered in which reranker model to run. If you go the other way and run separate processes on one card, the isolation mechanism becomes the next decision, and the trade-offs are in sharing one GPU between processes. Infinity also documents an offline Python engine through AsyncEngineArray.from_args and an await engine.embed(...) call, which removes the HTTP hop entirely for a batch job that does not need a server.

Second, vLLM's pooling runner is the consolidation option, but be exact about what it consolidates. Each vLLM instance supports one model runner, so an embedder in pooling mode is still a separate instance from the one generating tokens. What you save is a second piece of software in the estate: the same image, the same flag vocabulary, the same metrics endpoint, the same vendor review you already completed. Since the docs decline to call it faster, that is the entire case, and it is a real one where a new server means a new approval.

Third, TEI ships the operational surface a regulated deployment needs anyway, without a sidecar: --api-key for Bearer-token authorization, --json-output for structured logs, --otlp-endpoint for OTLP over gRPC, and --prometheus-port defaulting to 9000. If your control framework requires authenticated internal service calls and structured audit logs, that is a real differentiator against assembling the same things around a Python server yourself. Weight resolution with no outbound network is a separate problem with its own answer, covered in running vLLM with no outbound network.

The other on-prem cost is silent truncation. When the corpus cannot leave the building, you cannot diff your vectors against a hosted provider's to find out that a slice of your index came from partial chunks, and re-embedding two million chunks costs the 8.9 hours above plus the coordination to schedule it. Letting 422s pile up during the run is cheaper.

Start with TEI if your checkpoint is on its architecture list, move to Infinity when it is not or when you need a reranker and an embedder in one process, and treat the vLLM pooling runner as a consolidation move rather than a performance one. Where this tier sits in the wider retrieval stack is mapped in our RAG systems pillar.

This week, do the cheap check. Run a few thousand chunks from your real corpus through TEI's /tokenize route and count how many come back above your model's cap, then restart the server with --auto-truncate false and re-run the same sample. The gap between the two counts is how many vectors in your index were built from partial chunks. Until you have that number, the retrieval quality argument is about the wrong thing.