A context vector is a numerical representation, usually in the form of a vector (an array of numbers), encoding the semantic content and substance of an input sequence of data such as a sentence or a paragraph. In classical encoder-decoder architectures this vector is the final output of the encoder that has processed the whole input sequence. It served as the single, condensed source of information passed to the decoder for generating the output. This fixed-length conception, however, constituted an information bottleneck, which limited model performance on longer sequences. In modern architectures the concept has largely been replaced by attention mechanisms, which allow dynamic access to information from the entire input sequence.
Imagine reading a sentence. What stays in your mind is not just the individual words but above all its overall sense – the main idea. A context vector is precisely this “idea”, but in a form intelligible to an artificial intelligence. Once the model has “read” the input text, it condenses its complete meaning into a single package of numbers. That package then serves as the basic building block for further tasks, such as producing an accurate translation into another language.