Dropout is a regularization technique in which a certain proportion of the neurons in a layer are switched off at random during every training step – their output is set to zero. The proportion of disabled units is a hyperparameter, typically between ten and fifty percent, and the mask is drawn afresh for every batch. The remaining activations are rescaled so that the expected sum is preserved. The effect is twofold. First, the network cannot rely on any particular unit, because it may vanish at any moment, and so it has to build redundant, more robust representations; this guards against co-adaptation, where a group of neurons learns to work only in tandem. Second, dropout can be viewed as a cheap way of training an exponentially large ensemble of subnetworks whose predictions are implicitly averaged at inference – conceptually close to bagging. At inference dropout is switched off and the full network is used. It suppresses overfitting effectively, although in modern transformers trained on giant corpora its importance declines and it is sometimes set to zero.
Imagine a sports team whose coach sends a third of the squad to the bench at random before every training session. Nobody knows in advance who it will be. The team therefore cannot lean on a single star, because that star may not be on the pitch, and every player is forced to learn more than one role. Then, when the real match comes, the full line-up takes the field – and it is noticeably more resilient than if it had spent the entire preparation in the same fixed formation. This is exactly how dropout stops a network from memorizing the training data through a handful of privileged connections.