ALGOTECHNIC
All articles
8 min read

Mixture-of-Experts in Production: Sparse Compute, Dense Operational Constraints

How top-k routing, expert capacity, communication and total-weight memory determine whether sparse MoE models deliver useful production throughput.

  • mixture of experts
  • model serving
  • distributed systems
Tokens pass through a sparse router to selected experts while a full bank of model weights remains resident below.

A mixture-of-experts (MoE) layer replaces one feed-forward network with many and evaluates only a small subset for each token. This separates model capacity from per-token expert computation: adding experts can add parameters without adding their full cost to every token. Mixtral 8x7B makes the distinction concrete. Its router chooses two of eight feed-forward experts at each layer; the model contains 47B parameters, of which 13B are active for a token.

Those 13B active parameters are not a memory footprint or a latency promise. They describe one token’s computation path. A service must store or otherwise supply the other weights, move tokens to the selected experts and combine the results. The production case for MoE therefore rests on a system question: does the quality gained through conditional computation repay the routing, communication and capacity costs on the actual workload?

Top-k routing is a weighted computation

In Mixtral’s formulation, a learned router produces one logit per expert for token representation x. Top-k selection retains the highest-scoring k experts; a softmax over those logits produces their gate weights. With selected set T, the layer computes y = Σᵢ∈T gᵢ(x)Eᵢ(x). Experts with zero gates do not run. This is conditional computation inside one jointly trained model, not a request-level choice among independently trained models.

Switch Transformer uses k = 1, multiplying the selected expert’s output by its gate value. That simplifies dispatch and reduces expert work and communication relative to top-2 routing. Mixtral uses k = 2 and adds the two weighted SwiGLU outputs. Increasing the number of experts while holding k fixed grows total parameters without a proportional rise in expert FLOPs. Increasing k, by contrast, adds expert evaluations and more assignments to transport.

The router makes a fresh decision at every MoE layer and token position. Expert names such as mathematics or coding therefore describe hopes, not contracts. Mixtral’s routing analysis found no obvious topic-based partition across several domains, while showing syntactic patterns and repeated choices between consecutive tokens. Operators should treat these as learned numerical pathways. They should not infer clean subject boundaries or assume that traffic categories will distribute evenly.

Capacity converts skew into overflow or padding

Dynamic routing meets a static implementation constraint when each expert receives a fixed-size buffer. For top-1 Switch routing with T tokens and E experts, capacity is (T / E) × capacity factor. The factor adds headroom above the uniform share. If assignments exceed the buffer, Switch skips expert computation for overflowed tokens and passes their representations through the residual connection. Unused slots are padded.

Neither extreme is free. More capacity reduces overflow but reserves memory and can spend computation and communication on padding. Less capacity improves buffer occupancy but makes skew more damaging. This is not merely a throughput problem: bypassing an expert changes the model computation for that token. Production evaluation must use the same overflow policy as deployment rather than measuring quality with an ideal, unlimited router.

Training objectives can discourage this skew. Switch’s auxiliary loss combines the fraction of tokens dispatched to each expert with the mean router probability assigned to it, encouraging both vectors towards uniformity. GShard similarly uses an auxiliary loss, expert limits and probabilistic second-expert routing. These mechanisms shape training; they do not guarantee a balanced next serving batch. A language mix, prompt pattern or burst of similar tokens can still concentrate work.

Three tokens enter a top-2 router. Selected tokens cross device boundaries to weighted experts with finite capacity, then return for a weighted sum, while all expert weights remain resident.
Top-k routing limits expert computation per token, not the weight store. Expert parallelism adds dispatch and return traffic; finite buffers turn uneven assignments into overflow or padding.

Measure assignments by layer and expert, not just aggregate router entropy. Track admitted tokens, overflow, padding and the ratio between the busiest expert and the mean. Include skewed and bursty traffic in load tests, and check answer quality under those conditions. A good training-time average can hide a slow or lossy serving batch.

Expert parallelism makes routing a network operation

Expert parallelism distributes different experts across devices. Tokens begin grouped by request or data-parallel shard, then must be regrouped by expert. An all-to-all collective dispatches their representations; expert outputs cross the network again before restoration to token order and weighted combination. This happens at every distributed MoE layer. Top-2 routing doubles the expert assignments compared with top-1, although implementations can differ in what they transmit.

The trade-off changes with hardware and batch shape. More devices hold more weights and offer more aggregate memory bandwidth, but also widen the communication group. Small decode batches make message latency prominent; large prefill batches move more data and can make interconnect bandwidth prominent. Attention and other non-expert layers still require their own weights and parallelisation strategy. MoE does not make the whole model sparse.

DeepSpeed-MoE addresses this systems problem directly. It combines expert and tensor parallelism with a hierarchical all-to-all, coordinates communication with the chosen parallel layout and fuses routing-related kernels. The lesson is broader than those optimisations: device count, network topology and parallel layout belong in the serving design. Adding GPUs can reduce per-device weight pressure while increasing collective cost, so the fastest configuration need not be the largest one.

Kernels decide whether sparsity becomes throughput

Accelerators favour large, regular matrix multiplications. Under balanced routing, T tokens using k of E experts leave roughly kT / E tokens per expert. Raising expert count fragments the batch. Each expert then performs a smaller matrix operation, potentially leaving compute units idle and paying fixed kernel and weight-loading costs. Sparse FLOPs describe avoided arithmetic, not the efficiency of the work that remains.

Batching requests can restore useful expert batch sizes. It also changes the latency budget: waiting for companions adds queueing delay, and a diverse batch may activate nearly every expert. Prefill and decode consequently need separate measurements. A configuration that sustains high throughput with many concurrent prompts need not minimise the time between generated tokens for one response.

Kernel design provides another lever. MegaBlocks expresses expert computation as block-sparse matrix products with variable-sized token groups, avoiding token dropping and full fixed-capacity padding. It still constructs routing metadata, permutes tokens and pads groups to block boundaries. Dropless does not mean overhead-free. DeepSpeed-MoE instead uses a dense token-to-expert mapping table, fuses its gating function and replaces sparse einsums with data-layout transformations. Both show why an MoE serving benchmark must include dispatch, permutation and recombination rather than timing expert matrix multiplication alone.

Total weights govern the memory plan

Every expert’s weights must be available when routing selects it. Mixtral’s paper accordingly ties serving memory to its 47B total parameters, while reserving 13B for the active path. A single token touches only part of that store; a batch can collectively touch much more. DeepSpeed-MoE describes the same best-case and worst-case split between one token’s critical path and the aggregate parameters needed by a batch.

Distributing the full weight store increases aggregate device memory and bandwidth. It does not reduce the checkpoint to its active subset. Attention weights, key–value caches, activations, router state and communication buffers also consume capacity. Quantisation can reduce weight bytes, but its realised speed depends on supported kernels and dequantisation, and its quality needs separate evaluation.

Memory bandwidth can dominate small-batch inference even where arithmetic has fallen. Conversely, larger batches improve arithmetic intensity while increasing the range of experts read and the volume routed. Active parameters remain useful for estimating one term of compute. They cannot stand in for resident bytes, bytes read per batch or end-to-end latency.

Evaluate the serving system, not the parameter slogan

Start with a dense model of acceptable task quality as the control. Compare complete serving stacks on the same hardware budget, with representative prompt lengths, output lengths and request mixtures. Measure time to first token and subsequent token latency separately, report tail latency alongside throughput, and record peak memory. Keep batch size visible: it is an operating decision with a queuing cost, not a free normalisation.

Then expose the MoE-specific terms. Profile router and permutation kernels, both communication directions, expert utilisation and time spent waiting for the busiest device. Record overflow and padding by layer. Repeat the comparison under skewed traffic and at the intended concurrency, because balanced synthetic batches conceal both network stragglers and capacity failures.

MoE offers a useful bargain when additional total parameters buy quality that a practical dense alternative cannot reach at the same serving cost. The bargain fails when weight residency, fragmented batches or communication consume the nominal compute saving. Sparse activation is the architectural opportunity; measured system behaviour decides whether production keeps it.

Sources and further reading