Self-Host Kimi K3: GPU Sizing for 1,561 GB of Weights

Do not take a size figure from a blog post, including this one. The model repository exposes blob sizes through its API, and summing the safetensors shards takes one command:

curl -sL "https://huggingface.co/api/models/moonshotai/Kimi-K3?blobs=true" \
 | python3 -c "
import json,sys
d=json.load(sys.stdin)
s=sum(f['size'] for f in d['siblings'] if f['rfilename'].endswith('.safetensors'))
print(f'{s:,} bytes = {s/1e9:.1f} GB = {s/2**30:.1f} GiB')"

Run on 2026-07-28, that returns 1,560,936,091,448 bytes across 96 shards, out of 118 files in the repository. That averages 16.3 GB per shard, which matters later when you budget the cold start. In the three units people actually argue in: 1,560.9 GB decimal, 1,453.7 GiB binary, 1.42 TiB.

That fourth row deserves attention because it is in the official recipe, not a content farm. The YAML carries a code comment stating the value is a pre-release estimate of 2.8T x 0.5 bytes x 1.2 headroom, with an instruction to replace it once weights are published. As of 2026-07-28 it has not been replaced. Anyone provisioning to vram_minimum_gb: 1680 has 119.1 GB left after weights, which does not cover KV cache, recurrent state, activations, and CUDA graphs at any concurrency worth serving.

The model's config.json carries a compressed-tensors quantization block in mxfp4-pack-quantized format with num_bits: 4, strategy: group, group_size: 32, and scale_dtype: torch.uint8. That last pair is the whole story on size. Every block of 32 weights carries one uint8 shared exponent, so the amortized cost is 4 + 8/32 = 4.25 effective bits per weight.

Run the arithmetic all the way through:

The 73 GB residual is the second half of the explanation, and it is the part that changes how you think about the model rather than just how you size it. The quantization config carries an explicit ignore list:

"ignore": [
  "re:.*self_attn.*",
  "re:.*shared_experts.*",
  "re:.*mlp\\.(gate|up|gate_up|down)_proj.*",
  "re:.*lm_head.*",
  "re:.*vision_tower.*",
  "re:.*mm_projector.*"
]

MXFP4 covers the 896 routed experts and nothing else. Everything on the always-on path stays BF16: all self-attention across all 93 layers, both shared experts (which fire on every single token), the dense MLP projections in the one dense layer, the LM head over a 163,840-row vocabulary, and the entire vision tower and multimodal projector.

This kills a claim that circulated widely in launch week: that MXFP4 reduces memory bandwidth requirements by roughly 4x versus FP16. That is false for the per-token hot path. Every token reads 2 shared experts and 93 attention blocks at full BF16 width. Only the 16 routed experts selected per token get the 4-bit read. Treat the bandwidth win as applying to a fraction of the traffic, and treat --language-model-only as a real optimization rather than a convenience flag: on a text-only workload the BF16 vision tower is dead weight occupying HBM you paid for.

Here is the sizing rule nobody has written down, and it inverts the habit you built serving dense transformers.

Only 24 of the 93 layers grow with context. Those use multi-head latent attention, storing kv_lora_rank + qk_rope_head_dim = 512 + 64 = 576 elements per token per layer. Across 24 layers that is 13,824 elements per token:

bytes_per_token_bf16 = 13,824 * 2 = 27,648 bytes = 27.0 KiB

A full million-token sequence costs 13.5 GiB at FP8. On a node with 2.3 TB of HBM that is noise. Compare that to a conventional 70B dense model with grouped-query attention: 80 layers x 8 key-value heads x 128 head dimensions x 2 tensors x 2 bytes is 320 KiB per token, so 128K of context costs roughly 40 GiB for a single sequence. K3 carries the same 128K for 3.4 GiB at BF16, and that gap is how far the economics have moved. If that per-token formula is unfamiliar territory, our walkthrough of vLLM KV cache OOM errors covers the general equation and the flags that move each term.

The other 69 layers are the catch. Linear-attention layers carry a recurrent state that is fixed per sequence and does not grow with context at all: 69 layers x 96 heads x 128 x 128 = 108,527,616 elements, plus a short-convolution state of 69 x 96 x 128 x 4 = 3,391,488 elements.

Which of those two rows applies is not documented. Delta-rule kernels commonly hold recurrent state in FP32 for numerical stability, and we have not confirmed what vLLM does on device for this model. Budget the FP32 row and be pleasantly surprised, rather than the reverse.

The operational consequence: every concurrent request costs a flat 213 to 427 MiB whether the prompt is 200 tokens or a million. Long context is cheap on K3, batch size is expensive. That is why the official recipe hard-caps --max-num-seqs at 32 on CUDA and allows 128 on ROCm. If your capacity model assumes memory scales with tokens in flight, it will be wrong here in a way that only shows up under load.

Per-GPU capacities above are vendor-published specifications. H200 at 141 GB is well established; confirm the 288 GB figures against the datasheet for the exact SKU your vendor is quoting before signing anything, since capacities have moved between announcement and shipping SKUs in this generation.

Hopper pays twice. The first cost is node count: two 8-GPU nodes minimum, with all the cross-node interconnect complexity that implies. The second is the kernel. The official recipe forces --moe-backend marlin on Hopper, meaning there is no native MXFP4 mixture-of-experts kernel there and the routed expert weights are dequantized on the fly. Blackwell runs MXFP4 natively on the MoE path. The recipe marking H200 as verified hardware means "this configuration was tested," not "eight of these are enough."

Published throughput at batch size 1, measured by the vLLM team on Blackwell-class hardware:

Note what is missing: no time-to-first-token figures, no throughput at concurrency above 1, no cost per million tokens, no cold-start timings. None of those have been published, and you should not trust anyone who prints them one day after release. Build your own numbers before committing capacity.

Minimum version is 0.27.0 and the recipe flags nightly as required. Prebuilt images exist for both CUDA and ROCm. Four base arguments apply on all hardware: --trust-remote-code, --load-format fastsafetensors, --moe-backend auto, --gpu-memory-utilization 0.95.

The recommended production profile on Blackwell:

export VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1
export VLLM_ALLREDUCE_USE_FLASHINFER=1
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_USE_RUST_FRONTEND=1

vllm serve moonshotai/Kimi-K3 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --moe-backend auto \
  --gpu-memory-utilization 0.95 \
  --no-enable-flashinfer-autotune \
  --max-model-len 1000000 \
  --kv-cache-dtype fp8 \
  --attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}' \
  --max-num-batched-tokens 32768 \
  --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

Three flags there are easy to drop and expensive to omit. --enable-prefix-caching is required because prefix caching is off by default, which is the opposite of what most operators assume. use_prefill_query_quantization inside --attention-config is required whenever you set --kv-cache-dtype fp8. And --max-model-len 1000000 is only sane because of the KV math above; on most models that flag would be reckless.

Hopper overrides --moe-backend marlin, adds --disable-custom-all-reduce, and keeps --no-enable-flashinfer-autotune. AMD diverges more than you would expect: --load-format auto rather than fastsafetensors, --max-num-seqs 128 rather than 32, --max-num-batched-tokens 4096 rather than 32768, --mm-encoder-tp-mode data, and a required environment variable VLLM_USE_BREAKABLE_CUDAGRAPH=0 because the ROCm build auto-enables it to 1 and that breaks. Do not port a CUDA command line to ROCm by changing the image tag.

For speculative decoding, which is where the 3x per-user throughput comes from:

--max-num-seqs 32 \
  --speculative-config '{"model":"Inferact/Kimi-K3-DSpark",
    "num_speculative_tokens":7,
    "method":"dspark",
    "attention_backend":"FLASHINFER_MLA",
    "draft_sample_method":"probabilistic",
    "rejection_sample_method":"block"}'

Speculative decoding needs additional VRAM, hence the cap on concurrent sequences. On ROCm swap attention_backend to TRITON_MLA, since the FlashInfer backend is CUDA-only.

Separating prefill from decode into distinct pools is supported and is where the recipe gets genuinely opinionated: tensor-expert parallelism with tp=8 and --enforce-eager on the prefill side, data-expert parallelism with tp=1, --moe-backend deep_gemm_mega_moe, --max-num-seqs 32, and --no-enable-prefix-caching on the decode side. Use --all2all-backend deepep_v2 for RDMA fabrics and flashinfer_nvlink_one_sided for NVLink. If you have not run a split-pool deployment before, our post on prefill-decode disaggregation at serving scale covers why the two phases want different hardware profiles in the first place.

What not to ship: the model card's own launch instructions are pip install vllm followed by vllm serve "moonshotai/Kimi-K3". No tensor parallelism, no --trust-remote-code, no memory configuration. That will not start on any real cluster.

This is the checkpoint that stops procurement, and almost nobody covering the release has read the file.

The license in the repository is titled "Kimi K3 License," copyright 2026. It is not MIT, not Apache 2.0, and not a modified MIT. Our own comparison published on 2026.07.21 described it as modified MIT with an attribution clause based on pre-release statements. That was wrong, and the released text is materially different.

Three clauses matter:

Section 2 is the commercial gate. It defines "Model as a Service" as giving a third party access to inference or fine-tuning in a manner allowing meaningful control over the inputs, parameters, or training data. If you operate such a service and the aggregate revenue of you plus affiliates exceeds USD 20 million over any consecutive 12 months, you must enter a separate agreement with the publisher before using the software for any commercial purpose. Two carve-outs sit inside it: end-user products with model capabilities embedded solely within specific features or harnesses, and mere relaying of requests to models hosted by others.

Section 3 is attribution. Products or services above 100 million monthly active users, or above USD 20 million in monthly revenue, must display "Kimi K3" prominently in the user interface.

Section 4 is the exemption that matters for regulated buyers. Sections 2 and 3 do not apply to internal use, defined as any use that does not make the software, its outputs, or its underlying capabilities available to third parties. A bank, insurer, or hospital running K3 on-prem for its own staff is fully exempt from the MaaS clause regardless of revenue. That is the single most useful sentence in the document for the audience most likely to want the weights, and it is exactly the sovereignty argument covered in our comparison of cloud versus on-premise AI security and cost.

The practical procurement sequence: have counsel read sections 2 through 4 and write down which side of the internal-use line each planned deployment falls on, before the hardware purchase order rather than after. Section 5 is a standard as-is warranty disclaimer, which for a regulated deployment means the entire model risk burden sits with you.

Work these in order before anyone approves a budget.

The honest state of things one day after release: nobody has published a measured production deployment. The community threads on the model repository contain estimates ranging from 8x H100 to 35x H100 with no running system behind any of them, and the official recipe still ships a pre-release VRAM estimate its own code comment asks someone to replace. Measure the checkpoint, budget the state, read the license. That gets you further than every hardware guide currently ranking for the query.

Sizing on-premise inference for regulated environments is the work our model-serving practice exists to de-risk, and the first deliverable is always the same: a defensible number instead of a vendor estimate.