Cross-attention is a variant of the attention mechanism in which the queries come from one sequence while the keys and values come from another. That is what distinguishes it from self-attention, where all three projections derive from the same input. In the classic encoder-decoder architecture, cross-attention lets the decoder “look” at the entire representation of the input built by the encoder as it generates each output token, and pick from it exactly what it needs at that moment. Historically this is what removed the bottleneck in machine translation, where an entire source sentence previously had to be squeezed into a single context vector. Today cross-attention is a basic tool of multimodal models: a text decoder uses it to read a representation of an image, audio or video. Diffusion models for image generation use it too – the denoising network attends through cross-attention to the embedding of the text prompt, which is how the resulting picture comes to follow the brief.
Picture an interpreter rendering a speech into another language. They do not memorize the whole address and translate it blind; they have the speaker’s notes in front of them and, with every word they utter, glance at them deliberately – now for a name, now for a figure, now for that awkward technical term. The notes are one sequence, their own speech is the other, and the glance between the two is precisely cross-attention. In the same way, a model describing a photograph looks at a different part of the image for each successive word: at the animal for “dog”, at the background for “grass”.