The vanishing gradient is a phenomenon in which the learning signal weakens exponentially on its way back to the first layers of a network during backpropagation, until it approaches zero. The cause lies in the chain rule: the gradient arises as a product of the derivatives of all the layers above a given layer, and if the typical derivative is smaller than one, a product of dozens of such factors collapses. In practice this means the lower layers barely learn at all, the network behaves as if it were far shallower, and its depth becomes a liability. Its mirror image is the exploding gradient, where the derivatives are greater than one and the product grows to astronomical values, causing wild jumps in the parameters and the collapse of training, often all the way to NaN. Historically this was the main obstacle to deep learning and the reason recurrent networks could not remember long dependencies. Today’s defence is a combination of measures: non-saturating activation functions such as ReLU, residual connections that give the gradient a direct route, layer normalization, careful weight initialization, and – against explosion – gradient norm clipping.
Imagine a message passed down through twenty levels of management, where every manager traditionally leaves half of it out. The chief executive says something important; what reaches the worker at the far end is noise carrying no information, so the worker changes nothing about the job. That is the vanishing gradient. The opposite case is a company where every manager doubles the message for emphasis – at the end the worker reads that everything must be torn down immediately, and duly does it. The fix is to let the chief executive talk to the lower levels directly, which is exactly what residual shortcuts do.