Expert Parallelism vs Tensor Parallelism for MoE Serving

Sparsity is the whole point of the architecture. Mixtral's paper describes each layer as eight feedforward blocks with a router that selects two of them per token, so each token has access to 47B parameters but uses 13B active ones (arXiv:2401.04088). DeepSeek-V3 scales the same idea: 671B total parameters with 37B activated for each token (arXiv:2412.19437). The published lineage goes back to GShard, which scaled a sparsely-gated MoE translation model beyond 600 billion parameters using automatic sharding (arXiv:2006.16668).

Open the checkpoints and the ratios get sharper. DeepSeek-V3's config.json carries n_routed_experts 256, num_experts_per_tok 8 and n_shared_experts 1 across 61 layers, the first 3 of which are dense (first_k_dense_replace 3). Qwen3-30B-A3B carries num_experts 128 and num_experts_per_tok 8 across 48 layers. One routed expert in 32 fires per token on the first model, one in 16 on the second.

Tensor parallelism was built for dense layers, where every weight participates in every token and splitting a matrix across ranks splits the work evenly. Point it at an expert layer and it does the same thing to all 256 experts: each rank holds a slice of every expert, and the layout carries no information about which experts a given token actually needs. That is serviceable on a small MoE. It is also a layout that has thrown away the one structural fact the model hands you.

Expert parallelism keeps that fact. The comment in vLLM's fused MoE configuration is blunt about the consequence: "In EP, each device owns a set of experts fully. There is no tensor parallel." The EP branch then sets tp_size=1 and tp_rank=0 for that layer.

The config field is one line with a one-line docstring, and both are worth reading literally:

enable_expert_parallel: bool = False
"""Use expert parallelism instead of tensor parallelism for MoE layers."""

MoE layers, and nothing else. The expert parallel deployment page's layer table has exactly two rows, expert layers and attention layers, and only the first one changes. A dense model has no expert layers for the flag to rewrite.

There is also no size to set. The expert parallel deployment page states that EP size is calculated automatically as EP_SIZE = TP_SIZE × DP_SIZE, and no --expert-parallel-size or --ep-size option exists in v0.28.0. You move EP size by moving tensor-parallel size, data-parallel size, or both, so the same EP 8 is reachable through --data-parallel-size 8 --tensor-parallel-size 1 or through --tensor-parallel-size 2 --data-parallel-size 4, which put attention in different places.

The gate that decides whether the flag does anything at all sits in fused_moe/config.py:

use_ep = (
    dp_size_ * pcp_size_ * tp_size_ > 1
    and vllm_parallel_config.enable_expert_parallel
)

On a single card, with tensor, prefill-context and data parallel sizes all at 1, the product is 1 and --enable-expert-parallel is inert. No warning, no error, no effect. If you are debugging why the flag changed nothing on a workstation, that is why.

The last row applies only at load time. enable_ep_weight_filter defaults to False, and its docstring says it skips non-local expert weights during model loading when expert parallelism is active, so each rank reads only its own expert shard from disk, which "can drastically reduce storage I/O for MoE models with per-expert weight tensors". It has no effect on 3D fused-expert checkpoints or non-MoE models. On network-attached storage feeding a multi-node load, that is a rollout-window question.

vLLM's own framing is that for MoE models, particularly those using multi-head latent attention, "it can be advantageous to use data parallel for the attention layers and expert or tensor parallel (EP or TP) for the expert layers". The two axes are set independently and then multiplied.

Start from the default, which is what you get if you configure data parallelism and stop there. DP 4 and TP 2 on one 8-GPU node, expert layers in a tensor-parallel group of size 8:

vllm serve $MODEL --data-parallel-size 4 --tensor-parallel-size 2

Add the flag and the same 8 GPUs become an EP 8 group, with attention still sharded TP 2 inside each of the 4 DP groups. The layout vLLM's DeepSeek example uses instead pushes TP down to 1, which replicates attention across all 8 ranks and gives the experts the whole node:

vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --tensor-parallel-size 1 \
    --data-parallel-size 8 \
    --enable-expert-parallel

The documentation prints that command with comments after the line-continuation backslashes, so each backslash escapes a space rather than the newline and the pasted command will not run. Copy the version above.

The fused MoE config's own docstring shows what that does to the two axes. At TP 2, DP 2 and EP enabled across four devices, every device reports TP = {1, 0}, DP = {2, rank} and EP = {4, rank}, with the comment that there are 2 engine instances and the experts are split between the 4 devices. Tensor parallelism inside the expert layer is gone, flattened into one expert group spanning both engines.

The cost that the flag's one-line docstring does not mention is stated plainly two pages away: "the data parallel ranks are not completely independent. Forward passes must be aligned, and expert layers across all ranks are required to synchronize during every forward pass, even when there are fewer requests to be processed than DP ranks."

The mechanism follows: "when any requests are in progress in any rank, we must ensure that empty 'dummy' forward passes are performed in all ranks that don't currently have any requests scheduled", handled by a separate DP Coordinator process plus "a collective operation performed every N steps to determine when all ranks become idle and can be paused".

Eight ranks, one request in flight, and seven GPUs are executing empty forward passes to keep the collective aligned. The ranks pause only once the coordinator's periodic collective finds all of them idle, so a trickle of traffic keeps the whole group awake.

No source we opened publishes the concurrency at which that stops mattering, and we are not going to invent one. What the shape tells you is that the waste is largest when steady-state concurrency sits below the number of DP ranks, and that the arithmetic you need is per rank rather than per deployment: the documentation notes that --max-num-seqs applies per DP rank, so 32 across 8 ranks is 256 sequences in aggregate and each rank has to find 32 of them to stay busy.

The close gives the experiment in full: the gap between a request rate below your rank count and one well above it is the number your capacity plan depends on.

The training objective wants balanced routing. Production traffic does not deliver it. vLLM says so in the EPLB section: MoE models are typically trained so each expert receives a similar number of tokens, but in practice the distribution across experts can be highly skewed.

Skew costs more under expert parallelism than under tensor parallelism, because the expert layer is a collective: every rank waits for the rank holding the popular expert. The research literature handles the same problem at training time with a capacity bound. Switch Transformers defines expert capacity as the number of tokens each expert computes, "set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor", and describes the overflow behaviour directly: "If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection" (arXiv:2101.03961). DeepSeek-V3 took the other route, describing an auxiliary-loss-free strategy for load balancing in its technical report.

At serving time vLLM gives you two levers, and they have very different prices.

The first is free. expert_placement_strategy defaults to "linear", which places experts contiguously: with 4 experts and 2 ranks, rank 0 gets experts [0, 1] and rank 1 gets [2, 3]. Setting it to "round_robin" gives rank 0 experts [0, 2] and rank 1 experts [1, 3], and the docstring says this "can help improve load balancing for grouped expert models with no redundant experts". DeepSeek-V3 is exactly that shape, with n_group 8 and topk_group 4 in its config. Changing placement costs no memory at all.

The second is EPLB, the expert-parallel load balancer, which records per-expert load and periodically rearranges the expert-to-rank mapping. The idea and the name come from DeepSeek's own Expert Parallelism Load Balancer repository, created 26 February 2025, whose README describes a redundant-experts strategy that duplicates heavily loaded experts and attempts to place experts of the same group on the same node to reduce inter-node traffic. vLLM ships its own implementation behind --enable-eplb, configured through --eplb-config:

vllm serve Qwen/Qwen3-30B-A3B \
  --data-parallel-size 8 \
  --enable-expert-parallel \
  --enable-eplb \
  --eplb-config '{"window_size":1000,"step_interval":3000,"num_redundant_experts":2,"log_balancedness":true}'

window_size 1000 and step_interval 3000 are the shipped defaults, meaning 1000 engine steps of load history and a rebalance every 3000 steps. The other two are not: num_redundant_experts defaults to 0 and log_balancedness to false, because logging it adds communication every step. The metric is average tokens per expert divided by maximum tokens per expert, so 1.0 is perfect balance and the number falls as one expert pulls ahead.

Two of those lines are missing from vLLM's own --eplb-config example, and both are load-bearing. That example carries neither --enable-expert-parallel nor any parallel size, so ParallelConfig raises ValueError("enable_expert_parallel must be True to use EPLB.") on the first check and, once that is fixed, rejects the run on a second check that requires tensor, prefill-context or data parallel size above 1. Copied as printed, it does not start. The trap runs the other way too: setting num_redundant_experts without --enable-eplb raises rather than being quietly ignored.

Redundant experts are not free. The documentation gives the overhead as NUM_MOE_LAYERS * BYTES_PER_EXPERT * (NUM_TOTAL_EXPERTS + NUM_REDUNDANT_EXPERTS) ÷ NUM_EP_RANKS, and prices it for one model: for DeepSeek-V3, approximately 2.4 GB for one redundant expert per EP rank. That memory comes out of the same budget as the KV cache, which vLLM says out loud, warning that EPLB "may not be a good fit for memory constrained environments or when KV cache space is at a premium". The project also recommends --eplb-config '{"num_redundant_experts":32}' in large scale use cases so the most popular experts are always available, with no definition of large scale and no measurement attached; treat it as the project's recommendation and price it with the formula before adopting it. If that trade pushes you into an allocation failure, the vLLM KV cache OOM triage order is the right place to start rather than the EPLB config.

Expert distribution is one division. Each EP rank holds NUM_TOTAL_EXPERTS ÷ NUM_EP_RANKS experts by default, or (NUM_TOTAL_EXPERTS + NUM_REDUNDANT_EXPERTS) ÷ NUM_EP_RANKS with EPLB redundancy.

Both divide evenly, which is not an accident of the examples. vLLM's documentation gives the distribution formula and no remainder rule, and we did not locate the code path handling an indivisible expert count. Pick an EP size that divides your routed expert count exactly and treat anything else as untested.

The attention side does not move. Expert parallelism rewrites the expert layers and leaves the KV cache where it was, so whatever duplication your tensor-parallel size and kv-head count already produce is unchanged by the flag; the duplication factor and the flag that changes it are a separate decision on a separate axis. What data-parallel attention changes is the count: "Each DP engine has an independent KV cache", so a DP 8 layout carries eight caches rather than one, and the documentation adds that the benefit of prefix caching is maximised by directing prompts intelligently, which means routing a prompt to the rank already holding its prefix. Above a certain scale that routing stops being the engine's job, which is where the orchestration layer above vLLM enters the picture.

For a concrete weight-footprint worked example on a checkpoint too large for one node, our Kimi K3 GPU sizing walkthrough prices 1,561 GB of weights and the node count that follows.

Under tensor parallelism the expert layer's collective is an all-reduce over activations. Under expert parallelism it is a dispatch of tokens to the ranks that own their experts and a combine of the results back. That is why the backend is a flag rather than an implementation detail.

Two values in the code will waste your afternoon. pplx and naive are still accepted by the All2AllBackend literal, but ParallelConfig logs a warning that the backend "has been removed" and substitutes allgather_reducescatter. If a runbook you inherited sets either one, it has not been doing what the runbook claims.

Cross-node expert parallelism is a hardware question rather than a flag question, and DeepSeek's published numbers for DeepEP, its expert-parallel communication library (repository created 17 February 2025), show why. The test configuration is 8K tokens per batch at 7168 hidden dimensions with top-8 experts, FP8 dispatching and BF16 combining.

These are DeepEP kernel figures, logical bandwidth by the README's own note, and they say nothing about tensor parallelism. What they say is that the same collective gets roughly an eighth of the bandwidth once it leaves the node, and less again as the group widens from EP 8 x 2 to EP 8 x 4.

For a rack you own, that is the procurement conversation. You choose the NVLink domain size once, at purchase, and no flag recovers it afterwards. An EP group inside one node runs on a fabric you already paid for; an EP group spanning nodes makes your InfiniBand or RoCE fabric part of the model's forward pass rather than just a model-loading concern, so size the NVLink domain against the EP size your expert count wants before the GPUs arrive.

The host environment is the other on-prem surprise, and it is a change-control problem more than a technical one. Expert parallelism's kernels are not a pip install: vLLM delegates DeepEP setup to its own EP kernels guide and DeepGEMM to that project's instructions, and the documented failure modes all sit below the application. A cannot register cq buf error on InfiniBand or RoCE means host and pod need ulimit -l unlimited. An init failed for transport: IBGDA error means the InfiniBand GDA kernel modules are missing, and the fix is a driver configuration script on every GPU node followed by a reboot. On Kubernetes, vLLM's guidance is that every pod runs with hostNetwork: true and securityContext.privileged: true to reach InfiniBand. In a regulated data centre, privileged containers with host networking and a per-node reboot are a security exception and a maintenance window, and both belong in the plan before anyone benchmarks a backend. One environment variable also prevents a startup hang on InfiniBand clusters, by forcing torch distributed discovery onto Ethernet:

export GLOO_SOCKET_IFNAME=eth0

Across two nodes the shape is the single-node layout plus addressing. Ranks 0 through 7 run on the primary, ranks 8 through 15 launch headless on the second node and point at the primary's address and RPC port, and EP size becomes 16:

# Node 1, primary, serves the HTTP endpoint
vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --all2all-backend deepep_low_latency \
    --tensor-parallel-size 1 \
    --enable-expert-parallel \
    --data-parallel-size 16 \
    --data-parallel-size-local 8 \
    --data-parallel-address 192.168.1.100 \
    --data-parallel-rpc-port 13345 \
    --api-server-count=8

# Node 2, headless worker
vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --all2all-backend deepep_low_latency \
    --tensor-parallel-size 1 \
    --enable-expert-parallel \
    --data-parallel-size 16 \
    --data-parallel-size-local 8 \
    --data-parallel-start-rank 8 \
    --data-parallel-address 192.168.1.100 \
    --data-parallel-rpc-port 13345 \
    --headless

If you have already separated prefill from decode into distinct pools, the backend choice is made for you: deepep_high_throughput on the prefill pool and deepep_low_latency on the decode pool, which is what vLLM's disaggregated architecture overview specifies. Whether to split at all is a different question, and prefill-decode disaggregation at serving scale works through the threshold where it pays off. Until you have split, leave the default backend alone: vLLM warns that the high-throughput and low-latency kernels are optimised for disaggregated serving and may show poor performance for mixed workloads.

Two numbers decide most of those rows, and both come off your own system. Your routed expert count decides which EP sizes are legal. Your steady-state concurrency against your DP rank count decides whether the synchronisation is amortised or wasted.

A recurring failure mode in serving deployments we have reviewed is not a wrong parallelism strategy but an unknown one: the expert layers were tensor-parallel by default while the runbook said "wide EP". The well-tuned throughput figures quoted for open-weight MoE models assume expert parallelism plus load balancing, a different stack from the one the flag alone gives you, and we walk that arithmetic in where to run open-weight models.

This week, do the cheap part first. Read the routed expert count out of the config.json you already have, and list the EP sizes that divide it:

python3 - <<'PY'
import json, sys
cfg = json.load(open(sys.argv[1] if len(sys.argv) > 1 else "config.json"))
total = cfg.get("n_routed_experts") or cfg.get("num_experts")
active = cfg.get("num_experts_per_tok")
shared = cfg.get("n_shared_experts", 0)
print(f"routed experts: {total}  active per token: {active}  shared: {shared}")
for ep in (2, 4, 8, 16, 32):
    if total and total % ep == 0:
        print(f"  EP={ep:<3} -> {total // ep} routed experts per rank")
PY

Then run the A/B on one node: the same checkpoint, the same trace, once with --data-parallel-size N and once with --data-parallel-size N --enable-expert-parallel, at a concurrency below N and again above it. That is two restarts and an afternoon, and it produces the expert-parallelism-versus-tensor-parallelism number that actually describes your workload.