Mixture of Experts (MoE) is an architectural pattern in which the dense feed-forward layer of a transformer is replaced by a set of parallel subnetworks – the experts – plus a small routing network (the router, or gating), which picks only a few of them for each individual token. Typically two experts out of eight, sixteen or several hundred are activated while the rest stay idle. This decouples the model’s total parameter count from the number of parameters actually used in a single pass: the model may hold hundreds of billions of weights, yet the computational cost corresponds to a mere fraction of them. The router is trained jointly with the rest of the network, and a balancing term is added to its loss function to stop all the traffic pooling into a few favourite experts while the others go unused. The practical limit is not computation but memory – every expert has to be loaded even though most of them will not be used, which leads to distribution across several accelerators. The approach was popularized by the Switch Transformer and Mixtral models and today underpins most top-tier language models.
Picture a large hospital. A nurse at reception spends no more than half a minute with each patient – just long enough to judge what the problem is and send them on to the right two specialists. The hospital employs a hundred doctors, but only a pair of them work on any one patient. If the whole team had to examine everybody, care would be extraordinarily slow and expensive. This way the hospital commands enormous collective expertise while the cost per treatment stays low. The drawback is obvious: all hundred doctors have to be in the building and on the payroll, even though most of them never see the patient in question.