An activation function is a non-linear transformation applied to the output of a neuron once it has computed the weighted sum of its inputs. Its presence is an existential condition for deep learning: composing any number of purely linear layers yields nothing but another linear mapping, so without non-linearity a hundred-layer network would have exactly the same expressive power as a single layer. Historically the sigmoid and the hyperbolic tangent were used, but they saturate for large input values and their derivative approaches zero, which leads to the vanishing gradient. The breakthrough came with ReLU, which simply clips the negative part to zero; it is computationally trivial, it does not attenuate the gradient in the positive region, and it naturally leads to sparse activations. Its weakness is “dead” neurons permanently stuck in the zero region, which the Leaky ReLU and ELU variants address. In today’s transformers the smooth functions GELU and SwiGLU have prevailed, as they do not kink sharply around zero and empirically deliver better results.
Think of it as a water valve on the neuron. The neuron adds up everything that has flowed in, and the valve decides how much to let through. If the valve were always wide open and passed on exactly what arrived, the whole network would be no more than a system of pipes – however elaborate, it would end up doing plain addition and multiplication and would never learn to tell a cat from a dog. A ReLU-type valve says: “If the inflow is negative, I shut completely; if it is positive, I open fully.” It is precisely this simple decision, repeated a million times across many layers, that gives the network the ability to describe even highly intricate relationships.