Recurrent neural networks (RNNs) process sequences element by element while maintaining an internal hidden state that is updated at every step from the new input and its own previous value. That state acts as memory: in theory a history of any length can be encoded into it, and the same set of weights is used at every time step, so the model copes with inputs of variable length. Training is done by backpropagation through time, where the network is unrolled into as many copies as the sequence has elements. And this is exactly where the fundamental weakness comes from: repeated multiplication by the same matrices makes the gradient either shrink exponentially towards zero or blow up, so the network is in practice unable to learn dependencies across longer gaps. The second limitation is enforced sequentiality – step n cannot be computed without step n−1, which rules out parallelization on modern accelerators. The gated LSTM and GRU variants substantially eased the first problem, but not the second. It was on precisely this count that the transformer architecture, which processes an entire sequence at once, overtook them in 2017.
Imagine someone reading a book through a slit so narrow that only one word is visible at a time. To keep the story coherent, they hold a single summary thought in their head and rewrite it after every word. It works surprisingly well for the last sentence or two. But what they learned on page ten will not survive two hundred pages of rewriting inside that one thought – it fades away. And because they have to read word by word, they cannot split the job with colleagues and get through the book ten times faster. Both are reasons why today’s models work differently.