Positional encoding is the mechanism that injects information about the order of elements in a sequence into a transformer’s input. The attention mechanism is in itself permutation-invariant – it treats the input as a set rather than a sequence, so without an additional signal the sentences “the dog bit the postman” and “the postman bit the dog” would be indistinguishable to the model. The original solution added to every embedding a vector composed of sinusoids at different frequencies, which lets the model derive relative distances as a linear function. Learned absolute positions followed, and today relative approaches prevail, above all rotary positional encoding (RoPE), which encodes position by rotating the query and key vectors in the complex plane. The advantage of RoPE is that the dot product of two vectors then depends only on the distance between them, so the model generalizes better to lengths it never saw during training. The way positional encoding is done is precisely what determines how far the context window can safely be stretched.
Imagine being handed an envelope with the words of one sentence cut out. You have all the words, but you have no idea what order they came in – and the meaning falls apart. Positional encoding is like a numbered stamp pressed onto each slip before they go into the envelope. Alongside a word’s meaning, the model always sees its place in the row as well. The cleverer variants do not stamp absolute numbers (“you are seventh”) but relationships (“you are three words behind this one”), which helps the model even when you hand it an envelope twice as long as anything it trained on.