The KV cache is an inference optimization for autoregressive models that stores the already computed keys and values of the attention mechanism for tokens processed so far, so that they do not have to be computed again. Without it, generating the n-th token would require an attention pass over the entire prefix, which makes the total cost quadratic in the length of the answer. With the cache, each step computes only one new key and one new value, appends them to memory, and compares the new query against the stored contents – turning the per-step cost linear. The price is memory: the size of the cache grows with the length of the context, the number of layers, the number of heads and the number of users served concurrently, and for long contexts it routinely exceeds the size of the model weights themselves. That is exactly why architectural adjustments such as multi-query and grouped-query attention appeared, letting several heads share the same keys and values, along with cache quantization and paging (PagedAttention). The KV cache also makes it possible to precompute a stable part of the prompt and reuse it repeatedly.
It is like keeping a running notebook of characters and events while reading a long novel. Without it you would have to leaf back through the entire book before every new page just to remember who is who – and reading would get slower with every page. With the notebook you simply add one line about what has just happened and glance at it once. You get through page seven hundred as fast as page two. The downside? The notebook keeps growing and taking up space on the desk – and when you are reading ten books at once, the desk runs out.