Unidirectional attention, also known as causal or masked attention, is a specific variant of the attention mechanism that restricts the flow of information to one direction only. When computing the contextual representation for a given element of a sequence, the model may look back only at the preceding elements and at the element itself, not at those that follow. Technically the restriction is implemented by a mask that effectively zeroes the attention weights for all future positions in the sequence. This approach is a basic building block of autoregressive models, especially decoders in the transformer architecture, since it guarantees causality – the prediction of a future element depends exclusively on information from the past.
Imagine writing a sentence word by word. When deciding what the fourth word will be, you can look at the first three words you have already written. You cannot look at the fifth or sixth word, because they do not yet exist. Unidirectional attention forces an artificial intelligence to work in exactly this way. While generating text it is allowed to “glance back” at the start of the sentence but strictly forbidden to “peek into the future” it is only now creating. This ensures that the model proceeds logically and does not invent things on the basis of information it would not have in a real situation.