{"id":10388,"date":"2026-09-13T11:56:02","date_gmt":"2026-09-13T09:56:02","guid":{"rendered":"https:\/\/www.kubicek.ai\/?p=10388"},"modified":"2026-09-13T11:56:19","modified_gmt":"2026-09-13T09:56:19","slug":"context","status":"publish","type":"post","link":"https:\/\/www.kubicek.ai\/en\/context\/","title":{"rendered":"Why a Language Model Forgets"},"content":{"rendered":"\n

In our Facebook group on artificial intelligence<\/a>, three questions came up within a single week that, at first glance, have nothing in common. A teacher wanted to produce a faithful copy of a twenty-page entrance exam and got broken output. Another member had 450 photographs of text, 800 MB in total across two ZIP archives, and could not upload them into ChatGPT or Claude at all. A third needed to analyse geological reports, each around 1,500 pages and over a gigabyte, and wrote that he “keeps hitting the limits”.<\/p>\n\n\n\n

Behind all three questions lies the same thing: the user loaded more onto the model than its working memory and interface can carry. That working memory has a name \u2013 the context window. So let me first explain what it is, why a model behaves worse when the window is full, where the limits lie for the models available in September 2026, and how to restructure a task so that it fits.<\/p>\n\n\n\n

What the context window is<\/h2>\n\n\n\n

A language model as such, at the moment it answers, has no memory in the human sense. Between two prompts it remembers nothing; it assembles every answer solely from what it has in front of it at that moment. What it has is the context: the provider’s system instruction, the entire conversation so far, attached files, tool results, possibly its own “reasoning”, and finally the answer it is currently generating. All of this is counted in tokens, where one token is roughly one English word. Czech text comes out at two to three tokens per word, worse than English, so the same content takes up more space in Czech.<\/p>\n\n\n\n

This is the place to pre-empt an objection I hear often: “But my ChatGPT does have a memory, it remembers things from other conversations.” It does remember \u2013 but it is not the model that remembers. The memory feature in ChatGPT and in Claude is part of the application that runs around the model, the wrapper, the interface. The application saves notes from your conversations into its own database and, with every new prompt, inserts selected notes at the beginning of the context, next to the system instruction. The model therefore “reads” them again with every answer, just like an attachment or previous messages, and just like those they take up room in the window. The same applies to referencing earlier chats, where the application searches the history for relevant excerpts and attaches them. From the model’s point of view it is all one input that came into existence a moment ago. The impression of persistent memory is created by the wrapper; the model itself remains memoryless and its blackboard stays the same size. One could say that the more notes the interface has written down about you, the less context remains for the actual task and the answer.<\/p>\n\n\n\n

The context window is the upper bound on how many tokens the model can hold at once. Picture a classic school blackboard: as long as there is room, you write. Once you have covered it completely, you cannot paint any more letters and have to start erasing. The comparison holds even in a detail that is often overlooked. A token that did not fit into the window does not exist for the model. The attention mechanism computes relationships only between tokens present in the input; whatever is not there has no representation the model could refer to. Overflowing the window is therefore not a mild drop in quality. The API returns an error, the chat application rejects the attachment or cuts off the beginning of the conversation without warning. Or, more often, it answers \u2013 but without knowledge of the complete input text.<\/p>\n\n\n\n

How attention works and why it is not free<\/h2>\n\n\n\n

To explain why a model behaves worse when the window is full, I need to describe at least roughly what it does inside. The transformer, the architecture underlying all of today’s large models, works with the attention mechanism [1]. Simplified: while processing each token, the model “asks itself” which of the other tokens in the context are relevant to it and assigns each of them a weight. Every token therefore looks at every other token.<\/p>\n\n\n

\n
\"Fig.
Fig. 1: Attention with a short (top) and a long (bottom) context. Every token is compared with all the others; the matrix on the right shows the weights of the highlighted token’s connections. As the sequence grows, the same attention is spread across more tokens and the difference between a relevant and an irrelevant connection blurs.<\/em>
<\/figcaption><\/figure>\n<\/div>\n\n\n

Two things follow from this. Computational cost grows with the square of the context length; doubling the context means four times as many pairwise relationships. This is why million-token contexts were expensive or unavailable for so long, and why some providers charge a higher rate above a certain threshold, typically 200\u2013270 thousand tokens \u2013 or, quite naturally, deliver lower accuracy in their answers.<\/p>\n\n\n\n

Attention is a finite resource that gets divided up. The more tokens compete for it, the smaller the share that falls to those that actually carry the answer. Semantically similar but irrelevant passages, so-called distractors, behave like noise that drowns out the relevant signal.<\/p>\n\n\n\n

On top of this comes positional encoding. The model has to know somehow where in the sequence a token sits. Position encodings are learned during training on certain lengths, and at distances the model did not see often enough during training their reliability drops. Vendors address this with various extension techniques, but none of them removes the effect entirely.<\/p>\n\n\n\n

Lost in the middle<\/h2>\n\n\n\n

The best-known empirical description of how a model behaves on a long context was given by Liu and colleagues in the paper Lost in the Middle<\/em> [2]. They tested models on tasks where the answer lay in one of many documents inserted into the context and systematically varied the position of the correct one. The result was a U-shaped curve: models did best when the relevant information sat at the beginning or the end of the context, and markedly worse when it sat in the middle. For some models, performance with the key information in the middle was worse than when they were not given the document at all. Another example to make it clearer. Imagine a student with limited time to work through notes on a book. He has no capacity to read the whole thing. What does he do? He will most likely read the introduction and the endings (or the back-cover summary). The model proceeds in a similar way.<\/p>\n\n\n

\n
\"Fig.
Fig. 2: The “lost in the middle” effect. The curve shows the model’s success rate depending on where in the context the information needed for the answer is located. Tokens at the beginning and end of the input (green areas) are used reliably; tokens in the middle (grey area, marked in red) receive the least attention and success drops to its minimum there. The resulting U shape was described by Liu and colleagues [2]; the practical lesson is that the most important instructions and data belong at the beginning or the end of the prompt, not in the middle.<\/em>
<\/figcaption><\/figure>\n<\/div>\n\n\n

For the teacher with the exam, this has a concrete consequence. If he pastes in twenty pages and wants a faithful copy, the tasks on page ten structurally receive less attention than the tasks on pages one and twenty. Output that breaks precisely in the middle is a property of the mechanism, and no wording of the instruction will change it.<\/p>\n\n\n\n

Subsequent research sharpened the picture. NVIDIA’s RULER benchmark [3] showed that the effective context, the length at which a model still maintains the quality it delivers on short inputs, is typically only 50\u201370 % of the nominal value. The NoLiMa paper [4] added that the needle in the haystack is easy as long as the fact can be found by literal word match; once a semantic match is needed and the question and the answer share no keyword, the performance of most models on long inputs falls sharply. And the Chroma Research report on “context rot” [5] documented that the degradation is gradual and begins long before the limit is reached. Models perform worse on the same task simply because the input is longer, even when the added text contains nothing relevant. A long conversation in which you have already corrected the output five times is therefore, in itself, a worse working environment than a fresh conversation with a clean prompt.<\/p>\n\n\n\n

What else plays a role<\/h2>\n\n\n\n

The input window is the most visible limitation, but far from the only one. In practice, work with large data also fails for reasons that relate to input length only indirectly, and users easily overlook them.<\/p>\n\n\n\n

Let me start with the output. A model may accept a million tokens on input, but on output it typically generates 64 to 128 thousand in a single response [11]. Generation is also autoregressive: each further token builds on the previous ones. With a long output, such as twenty pages of an exam in LaTeX, errors accumulate and towards the end the model tends to shorten, generalise or summarise instead of reproducing verbatim. I would say the teacher’s exam failed here rather than on the input.<\/p>\n\n\n\n

The second factor is the application layer, which the user usually knows nothing about. When you upload a large file into ChatGPT or Claude, the application by no means always inserts it into the context in full. For larger attachments it resorts to its own search over the document, and the model receives only selected excerpts. The user thinks the model read the whole PDF; in reality it saw ten fragments picked by a different algorithm. Add to this the loss of structure during conversion. A PDF with tables, columns and fractions collapses into linear text, and maps or charts disappear entirely unless they are processed separately as images. Moreover, depending on whether you have a paid or free account, and on the model type, the input context windows in the chat interface are usually smaller than what the model itself allows when you interact with it directly or through the API.<\/p>\n\n\n\n

The third is the ratio of instructions to data. A five-sentence prompt next to twenty pages of material becomes statistically negligible. The instruction is diluted in the context and the model drifts towards whatever is most abundant in the data rather than towards what you wanted. Related to this is contamination by the model’s own errors. Once a model produces a wrong output in a long conversation, that output is from then on part of the context and the model refers back to it in subsequent attempts. Correcting within one thread therefore turns out worse than a fresh start.<\/p>\n\n\n\n

The fourth is language. Czech tokenises into more pieces than English, so the same content takes up more of the window, and there is far less of it in the training data for long inputs. Degradation on long contexts is measurably steeper for non-English languages.<\/p>\n\n\n\n

And the fifth is the reasoning budget. In models with a reasoning mode, the reasoning counts towards the same window as the input and the output. A full input thus also eats into the space in which the model is supposed to solve the task.<\/p>\n\n\n\n

The limits in September 2026<\/h2>\n\n\n\n

So far we have discussed principles that hold. Now to the current state, which will age fastest and may no longer apply in six months. During the first half of 2026 all three major providers reached a context window of one million tokens. Google had it on Gemini since February 2024, Anthropic opened the million-token context on Claude Opus 4.6 in February 2026 and made it generally available in March, and OpenAI, after an interim step back to 256K, restored it with GPT-5.4 [7]. In summer 2026 Claude Sonnet 5 and Opus 4.8 also offer the million-token window at standard pricing, joined in July by Claude Opus 5; according to the Morph comparison, thirteen models including open-weight ones now offer a million tokens or more [11]. The number 1M itself has thus ceased to be a distinguishing feature.<\/p>\n\n\n\n

Reliability has become the distinguishing feature. The strictest publicly used measure today is the MRCR v2 benchmark in its eight-needle variant: the model has to find and reproduce verbatim eight specified pieces of information in a long text. Here the numbers diverge fundamentally. Claude Opus 4.6 reaches around 92\u201393 % at 256 thousand tokens and 76 % at one million, whereas its predecessor Sonnet 4.5 scored 18.5 % on the same test [6], [7]. GPT-5.6 Sol, according to independent measurement, holds 91.5 % in the 256K\u2013512K band and 73.8 % in the 512K\u20131M band [8]. Gemini 3 Pro at the time of measurement sat around 25 % at one million, although at 128 thousand the newer Gemini 3.1 Pro was practically level with Claude Sonnet 4.6 [7].<\/p>\n\n\n\n

Models that claim over 90 % accuracy at retrieving specific facts in the band around half a million tokens therefore do exist; they are the flagship models from Anthropic and OpenAI from spring and summer 2026. Even so, you cannot carelessly throw just anything into half a million tokens, and the reasons are in the footnotes. The figures for the vendors’ own models are mostly the vendors’ own measurements. The benchmark got harder during the year, so older and newer numbers are not directly comparable [12].<\/p>\n\n\n\n

What follows for the future<\/h2>\n\n\n\n

Whatever the numbers are a year from now, several principles will remain. Context is a budget, not a storage room; every token that does not carry information needed for the answer lowers the quality of the answer. Position in the context is not neutral; the beginning and the end are privileged, the middle is the most vulnerable. The type of task matters more than the length; retrieving a fact is orders of magnitude easier than aggregating, comparing or reproducing structure. Benchmark results are an upper estimate; effective context is always smaller than nominal. And output has its own limit, an order of magnitude smaller, which on long reproductions shows up before the input limit does.<\/p>\n\n\n\n

From these principles follow recommendations that apply across tools<\/strong>:<\/p>\n\n\n\n

    \n
  1. Split the task into parts so that each individual prompt works with a context well below the limit, including the output. <\/li>\n\n\n\n
  2. Place the most important instruction and the key data at the beginning, and if the prompt is long, repeat a brief recap of the request at the end. <\/li>\n\n\n\n
  3. Start a new conversation for every separate sub-task instead of continuing in one that already carries the baggage of previous attempts. <\/li>\n\n\n\n
  4. And finally, where accuracy matters, reach for a higher-tier model with active reasoning.<\/li>\n<\/ol>\n\n\n\n

    References<\/h2>\n\n\n\n

    [1] VASWANI, Ashish, SHAZEER, Noam, PARMAR, Niki et al. Attention Is All You Need. In: Advances in Neural Information Processing Systems 30 (NIPS 2017)<\/em>. Long Beach: Curran Associates, 2017, pp. 5998\u20136008.<\/p>\n\n\n\n

    [2] LIU, Nelson F., LIN, Kevin, HEWITT, John et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics<\/em>. 2024, vol. 12, pp. 157\u2013173. DOI 10.1162\/tacl_a_00638.<\/p>\n\n\n\n

    [3] HSIEH, Cheng-Ping, SUN, Simeng, KRIMAN, Samuel et al. RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv preprint<\/em> arXiv:2404.06654 [online]. 2024 [cited 2026-09-13]. Available from: https:\/\/arxiv.org\/abs\/2404.06654<\/a><\/p>\n\n\n\n

    [4] MODARRESSI, Ali, DEILAMSALEHY, Hanieh, DERNONCOURT, Franck et al. NoLiMa: Long-Context Evaluation Beyond Literal Matching. arXiv preprint<\/em> arXiv:2502.05167 [online]. 2025 [cited 2026-09-13]. Available from: https:\/\/arxiv.org\/abs\/2502.05167<\/a><\/p>\n\n\n\n

    [5] HONG, Kelly, TROYNIKOV, Anton and HUBER, Jeff. Context Rot: How Increasing Input Tokens Impacts LLM Performance<\/em> [online]. Chroma Research, 2025 [cited 2026-09-13]. Available from: https:\/\/research.trychroma.com\/context-rot<\/a><\/p>\n\n\n\n

    [6] ANTHROPIC. Claude Opus 4.6<\/em> [online]. 2026 [cited 2026-09-13]. Available from: https:\/\/www.anthropic.com\/news\/claude-opus-4-6<\/a><\/p>\n\n\n\n

    [7] YAGE.AI. Long Context Benchmarks: All Three Hit 1M \u2013 Now What?<\/em> [online]. 15 March 2026 [cited 2026-09-13]. Available from: https:\/\/yage.ai\/share\/long-context-benchmark-en-20260315.html<\/a><\/p>\n\n\n\n

    [8] TECH-INSIDER.ORG. Claude Fable 5 vs Opus 5 vs GPT-5.6 Sol<\/em> [online]. 2026 [cited 2026-09-13]. Available from: https:\/\/tech-insider.org\/claude-fable-5-vs-opus-5-vs-gpt-5-6-sol-2026\/<\/a><\/p>\n\n\n\n

    [9] SCOUT: Active Information Foraging for Long-Text Understanding with Decoupled Epistemic States. arXiv preprint<\/em> arXiv:2605.04496 [online]. 2026 [cited 2026-09-13]. Available from: https:\/\/arxiv.org\/abs\/2605.04496<\/a><\/p>\n\n\n\n

    [10] AIMULTIPLE. VELC-Bench: Verification on Long Context Benchmark<\/em> [online]. 4 August 2026 [cited 2026-09-13]. Available from: https:\/\/aimultiple.com\/ai-context-window<\/a><\/p>\n\n\n\n

    [11] MORPH. LLM Context Window Comparison (2026): 20 Models From 200K to 10M Tokens<\/em> [online]. 9 June 2026 [cited 2026-09-13]. Available from: https:\/\/www.morphllm.com\/llm-context-window-comparison<\/a><\/p>\n\n\n\n

    [12] R&D WORLD. Claude Opus 4.6 targets research workflows with 1M-token context window, improved scientific reasoning<\/em> [online]. 5 February 2026 [cited 2026-09-13]. Available from: https:\/\/www.rdworldonline.com\/claude-opus-4-6-targets-research-workflows-with-1m-token-context-window-improved-scientific-reasoning\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

    In our Facebook group on artificial intelligence, three questions came up within a single week that, at first glance, have nothing in common. A teacher wanted to produce a faithful copy of a twenty-page entrance exam and got broken output. Another member had 450 photographs of text, 800 MB in total across two ZIP archives, […]<\/p>\n","protected":false},"author":1,"featured_media":10391,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_seopress_titles_title":"","_seopress_titles_desc":"","_seopress_robots_index":"","_seopress_robots_follow":"","_seopress_robots_imageindex":"","_seopress_robots_snippet":"","_seopress_robots_primary_cat":"","_seopress_robots_breadcrumbs":"","_seopress_robots_freeze_modified_date":"","_seopress_robots_custom_modified_date":"","_seopress_robots_canonical":"","_seopress_social_fb_title":"","_seopress_social_fb_desc":"","_seopress_social_fb_img":"","_seopress_social_fb_img_attachment_id":0,"_seopress_social_fb_img_width":0,"_seopress_social_fb_img_height":0,"_seopress_social_twitter_title":"","_seopress_social_twitter_desc":"","_seopress_social_twitter_img":"","_seopress_social_twitter_img_attachment_id":0,"_seopress_social_twitter_img_width":0,"_seopress_social_twitter_img_height":0,"_seopress_redirections_value":"","_seopress_redirections_enabled":"","_seopress_redirections_enabled_regex":"","_seopress_redirections_logged_status":"","_seopress_redirections_param":"","_seopress_redirections_type":0,"_seopress_analysis_target_kw":"","_seopress_news_disabled":"","_seopress_video_disabled":"","_seopress_video":[],"_seopress_pro_schemas_manual":[],"_seopress_pro_rich_snippets_disable_all":"","_seopress_pro_rich_snippets_disable":[],"_seopress_pro_schemas":[],"footnotes":""},"categories":[10],"tags":[],"cat_tool":[],"class_list":["post-10388","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/posts\/10388","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/comments?post=10388"}],"version-history":[{"count":4,"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/posts\/10388\/revisions"}],"predecessor-version":[{"id":10396,"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/posts\/10388\/revisions\/10396"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/media\/10391"}],"wp:attachment":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/media?parent=10388"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/categories?post=10388"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/tags?post=10388"},{"taxonomy":"cat_tool","embeddable":true,"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/cat_tool?post=10388"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}