Autoregressive models are a class of generative model whose basic principle is predicting the next element in a sequence from all the preceding elements. Formally, such a model estimates the probability distribution for the next element conditional on knowledge of all previous elements of the sequence. Generation proceeds iteratively, step by step: the model first produces the first element, then uses it as part of the input for predicting the second, and repeats the process. This approach is central to tasks generating sequential data, such as producing text in language models or synthesising audio, and its operation requires strict observance of causality.
Imagine composing a long sentence. You do not utter it all at once but assemble it word by word, each successive word following logically from those before. An autoregressive model works exactly like such a careful narrator. It generates its output piece by piece, and in producing each new word it always glances back at everything it has produced so far. When it has written “Yesterday I went to the”, for instance, it looks at that history and infers that the most likely continuation is “cinema” or “office”. It then appends this new piece to the story and carries on.