Mixture of Experts: How Modern LLMs Scale Without Breaking the Bank

The performance of large language models scales with the number of parameters. More parameters mean better results but also exponentially higher compute costs, both to train and to run at inference. As models grow and daily usage increases, that cost curve becomes a serious problem. In a recent Passion Academy session, Ivo Kosa walked through Mixture of Experts an architectural approach that decouples model capacity from compute cost, and which underpins some of the most capable models in production today.

‍

Slides

‍

‍

The Core Idea

‍

A standard dense transformer processes every input through every parameter, every time. Every token passes through every layer, every weight is active on every forward pass. This is computationally clean but expensive and it gets more expensive linearly with model size.

‍

Mixture of Experts replaces the fully connected feed-forward layers in a transformer with a set of parallel expert networks and a router that decides which experts to activate for each token. Only a small subset of experts (typically two out of many) are activated per token per layer. The rest sit idle. Total parameter count goes up, but compute per forward pass stays roughly constant.

‍

The result is a model that can be much larger in terms of capacity without being proportionally more expensive to run.

‍

‍

Where the Idea Came From

‍

MoE isn't new. It dates back to a 1991 paper  ,"Adaptive Mixtures of Local Experts" by Jacobs et al, which used multiple specialist classifiers to differentiate spoken vowels. The intuition was straightforward: if a task naturally decomposes into subtasks, using separate specialised models and switching between them should stabilise training and improve generalisation compared to one model trying to handle everything.

‍

What that paper discovered is that the experts didn't need to be hand-designed. The router learned on its own which experts were useful for which inputs. Specialisation emerged through training rather than being imposed top-down. For a task with four classes, only two or three expert networks ended up being meaningfully used, suggesting each individual expert had more capacity than expected.

‍

‍

How It Works in a Transformer

‍

In a standard transformer, each layer has two components: an attention block that handles relationships between tokens, and a fully connected feed-forward network that processes each token independently. In a Mixture of Experts transformer, that second component (the feed-forward network) is replaced by a bank of expert networks plus a router.

‍

The router is a learned gating function. For each token, it scores every available expert and selects the top-k (typically two). Those experts process the token, their outputs are weighted by the router scores and combined and the result flows to the next layer. Random noise can be added to the routing scores during training to encourage the router to explore different experts rather than always defaulting to the same ones.

‍

The experts themselves can be any network architecture (i.e. FFN, attention, CNN) as long as all experts within the same layer produce an identical output shape so their outputs can be combined cleanly.

‍

‍

Why MoE Models Are Harder to Train

‍

Despite the efficiency gains at inference, MoE models are genuinely harder to train than dense models, for a few interconnected reasons.

‍

In a dense model, every parameter receives gradient updates on every training step. In a MoE model, only the selected experts receive gradient updates for any given token. Most experts see sparse, irregular updates which makes optimisation less stable and slower to converge.

‍

The router compounds this problem. It needs to learn which expert to use before the experts have learned anything useful... but the experts can't learn anything useful until the router is sending them relevant inputs. It's a chicken-and-egg problem that manifests as training instability, particularly early in the run.

‍

The most problematic failure mode is routing collapse where the router develops a strong preference for a small number of experts and consistently underutilises the rest. Ideally every expert should be activated roughly equally across a large batch of tokens. When this breaks down, you're paying the memory cost of all experts while only getting the compute benefit of a few.

‍

‍

The Trade-offs in Practice

‍

MoE models offer two genuine advantages:

‍

• Inference cost is significantly lower: only a fraction of the total network is active for any given token, which translates directly to faster and cheaper generation.
• comparable performance to dense models can be achieved at a fraction of the compute budget. This is why models like Gemma 4 and the Mixtral family have adopted the architecture.

‍

There are however limitations. For one, the whole network still needs to be loaded into memory, so the GPU memory requirement reflects the total parameter count, not just the active subset. This means MoE models are cheaper to run but not cheaper to host. Furthermore, fine-tuning on small datasets is problematic: sparse routing means most experts see very little data in a small training run, which leads to unstable and unreliable fine-tunes.

‍

Dense models still outperform MoE models of equivalent total parameter count on benchmarks. The MoE advantage is efficiency (comparable performance at lower compute) rather than raw capability.

‍

‍

The Broader Picture

‍

Mixture of Experts is one of the more elegant answers to the scaling problem. Rather than asking every part of the network to do every job, it asks different parts to specialise and lets training figure out what that specialisation should look like. The 1991 paper findings still apply as a collection of specialists, dynamically composed, is more efficient than a generalist doing everything at once.

‍

Interestingly, what's changed is the scale at which this is now being applied. The same principle that helped classify spoken vowels three decades ago is now being used to run trillion-parameter language models in production at inference costs that would otherwise be prohibitive. That's a meaningful line of continuity and a good reminder that foundational ideas in ML tend to resurface when the engineering catches up with the theory.

‍

References

‍

‍

• R.A.JACOBS ET AL, ADAPTIVE MIXTURES OF LOCAL EXPERTS
• W.FEDUS ET AL, SWITCH TRANSFORMERS: SCALING TO TRILLION PARAMETER MODELS WITH SIMPLE AND EFFICIENT SPARSITY
• GOOGLE, GEMMA 4, HTTPS://DEEPMIND.GOOGLE/MODELS/GEMMA/GEMMA-4/
• S.MU, S.LIN, A COMPREHENSIVE SURVEY OF MIXTURE-OF-EXPERTS: ALGORITHMS, THEORY, AND APPLICATIONS
• A.VASWANI ET AL, ATTENTION IS ALL YOU NEED

‍

‍