How does a mixture of experts work?

The Switch Transformer paper studied sparse routing as a way to scale a transformer: the model holds many experts, and a token is sent to a subset instead of through every feed-forward block.

Mixtral of Experts is a later, concrete language model on the same idea. Mixtral uses eight feed-forward experts in each layer.

For every token, a router picks two of them and adds their outputs. The selected pair can change from token to token.

Mistral’s paper says each token has access to 47 billion parameters and uses 13 billion active parameters during inference. Those two figures are the distinction to keep.

The Mixtral explainer stays with that model. This page is the general pattern.

What does “active” leave out?

Sparse activation can lower the compute per token. The serving system still has to be able to load the experts the router might choose.

Memory planning follows the checkpoint and the runtime, not the active-parameter headline alone.

Two numbers that are not interchangeable
FigureWhat it countsWhat to ask
Total parametersThe experts stored in the checkpointWill this file fit, and under which precision?
Active parametersThe experts used for one tokenWhat is the compute per token at your batch size?

Do experts check each other’s facts?

No. Experts are learned blocks of the network. Routing does not add a citation or a second opinion.

An answer can still be wrong, and a benchmark claim in a model paper is that paper’s result under its own tests.

Common questions

Is a mixture-of-experts model several separate models?

Not in the Mixtral design. It is one decoder-only model whose feed-forward block chooses among eight parameter groups.

Does a lower active-parameter count make the model cheap to run?

It describes compute per token in that design. It does not by itself tell you the memory required to hold the checkpoint.

Sources & further reading

  1. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity ↗
  2. Mixtral of Experts ↗
How this story was made

Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.

Our editorial standards
Back to all stories