A residual connection (also skip connection) is a shortcut that bypasses one or more blocks of a network and adds their input directly to their output. The block thus does not learn the whole desired transformation, only its deviation from the identity – the residual. The motivation was the empirical paradox that very deep networks achieved worse results than shallower ones, even though in theory it would suffice for the superfluous layers to learn to do nothing; it turned out that learning the exact identity is surprisingly hard for a non-linear layer. By adding the input, the identity becomes the default state for free. Equally fundamental is the effect on backpropagation: the derivative of a sum passes the gradient through the shortcut unchanged, creating a “gradient highway” and solving the vanishing gradient problem. The concept was introduced by ResNet convolutional networks and is today an indispensable part of every transformer block, where it is combined with layer normalisation. Without residual connections, training models with tens to hundreds of layers would not be practically feasible.
Imagine an editorial office where a manuscript passes through ten proofreaders. If each were allowed to rewrite the text from scratch, after the tenth round nothing of the original article would remain and the author would not recognise it. So a rule is introduced: each proofreader receives the text and may only append a list of his amendments. At the end, the original text and all the amendments are added together. If a proofreader has nothing to improve, he simply appends nothing – and the text passes through unchanged. What is more, when the author later wants to find out who spoiled what, he can follow a direct trail all the way back to the first proofreader without the information fading along the way.