Layer normalization is a technique for stabilizing the training of deep networks: for each individual sample it recomputes the activations so that they have zero mean and unit variance across the dimensions of the given layer. The result is then rescaled by two learnable parameters, so the network can recover whatever part of the original scale is useful. The aim is to suppress so-called internal covariate shift – the phenomenon where the distribution of inputs to each layer keeps changing during training because the weights of the layers below it are changing. Unlike batch normalization, layer normalization does not depend on the batch size or on the other samples in the batch, which makes it suitable for sequential data of variable length and for inference on a single input. In transformers its placement is crucial: the original architecture put it after the attention block, whereas today’s models move it in front (pre-norm), which markedly improves stability at large layer counts. A widespread simplified variant is RMSNorm, which drops the centring step and normalizes by the root mean square alone.
Imagine a school where every teacher marks with different severity. One hands out top grades to almost everybody; another gives middling marks for the same work. So the head teacher brings in a rule: everyone must rescale their marks so that the class average comes out the same. The ranking of pupils is preserved, but the numbers are finally comparable. That is exactly what layer normalization does inside a network – it prevents one layer from passing on absurdly large or vanishingly small numbers. Without it, a deep network would wobble during training and the run would often collapse after a few steps.