One successful attempt is all it takes for someone to believe that a language model can do maths. Then you upload a table of last month’s revenue, ChatGPT answers fluently, confidently, with a nicely formatted response. And the sum may well be wrong. Not by much, by a few per cent, just enough that nobody notices until they check it with an Excel formula. This scenario repeats itself over and over in practice, and behind it lies one widespread myth: that a model which understands text so well must surely understand the numbers in that text just as well.
It doesn’t. Not when it has to do the calculation itself, token by token, straight “in its head”. This clarification matters right at the start, because I will come back to it at the end: the author is not claiming that you should avoid numbers entirely when working with language models, but that you should not let yourself be lulled by an estimate derived from a language pattern. The solution is simple and it will be the main thing you take away if you read the article to the end. Before we get to it, though, let’s look at why the error actually happens. The reason is more interesting than it seems at first sight.
Numbers converted into numeric identifiers – tokens
A language model never sees text the way you see it. Before anything gets inside the network, it goes through so-called tokenization, a process that chops the input into smaller pieces, so-called tokens, and assigns each of them a numeric ID from the model’s vocabulary. With words this is fairly intuitive, but with numbers it starts to creak. The model converts numbers into numeric identifiers (IDs).
The number “12345” can fall apart into the tokens “123” and “45”, while „1 2345″ or „12,345″ gets chopped completely differently, even though it is the same value written with an extra space or comma. Different tokenizers cut numbers into one, two or three digits, and exactly how a given number falls apart depends on whether it stands alone, with a space, with a decimal separator or as part of a longer string. The researchers Singh and Strouse (2024) directly compared the approaches of individual model families: GPT-3.5 and GPT-4 have separate tokens for one-, two- and three-digit numbers and cut longer numbers from the left into groups of three, while LLaMA and PaLM tokenize every digit separately, and they demonstrated that precisely this choice measurably influences the model’s accuracy on arithmetic tasks.
The consequence is quite fundamental: the model has no guarantee that it will recognise that „742″, „0742″ and „742.00″ are the same value. For a human this is a trivial equivalence, for the model it is a question of whether this particular combination of tokens was sufficiently similar to something it saw during training. Yet arithmetic requires absolute consistency of representation – if you do not have it, you are not calculating, you are only estimating.
So that this does not stay at the level of a mere claim, you can verify it in the tokenizer provided directly by OpenAI at platform.openai.com/tokenizer. Into the field I entered five notations that a person reads without hesitation as the same value: “742”, “742.0”, “0742”, “00742” and “0000742.00” – that is, the number 742 on its own, with a decimal point and with a varying number of extra leading zeros.
The string has thirty-one characters in total and this particular tokenizer turned it into seventeen tokens, that is almost two tokens for every notation, and definitely not one consistent token for “this is the number 742” repeated five times. With a different model family or an older version of the tokenizer you get a different specific number, but the principle – that the same value in different notations gets a different number and different types of tokens – stays the same and can be verified on any of them.
What is interesting is that the same token ID appears only where the string of characters is absolutely identical – the token for the standalone “742” at the beginning is reused inside “742.0”, because there “742” is an exact match. But as soon as one or two extra zeros are added in front of the number, the tokenizer chops it completely differently and assigns completely different IDs, even though numerically it is still the same value with the same number of significant digits. Tokenization is therefore not governed by what the number means, but by which exact string of characters it has already seen and knows from its vocabulary.
Vectorisation: why an embedding ≠ a number line
Here we get to the heart of the matter, which most people are unaware of. After the text is chopped into tokens, each token is converted into a vector in a high-dimensional space. This step is called embedding, or vectorisation. The key question is: what exactly does this vector represent at the very input into the network? The answer is that it is not directly the mathematical value of the number on a number line, but the statistical and semantic context in which the given token appeared in the training data.
Put simply, at the level of the input embedding “742” and “743” are not close to each other because they differ by one – they are close or distant according to how similarly or dissimilarly they occurred in texts. The number “742” may have appeared mainly in the context of postcodes, while “743” in completely different situations, and the model then places them far apart in this initial embedding space, even though mathematically they are close neighbours.
That does not mean, however, that the model has no awareness of the magnitude of numbers. Kadlčík et al. (2025) showed that the embeddings of numeric tokens in pre-trained language models often already contain very precise information about the numeric value. This information need not be arranged linearly, though; the authors argue that it often has a sinusoidal/Fourier structure. When a probe with the corresponding inductive bias is used, the value of the number can be decoded from the embedding with high, almost perfect accuracy. Older, simpler probes, especially linear or ordinary regression ones, did not capture this structure and could therefore give the impression that models’ numeric representations are far less precise than they actually are.
A theoretical justification of why such structures appear at all is offered by the earlier work of Jiang et al. (2024): structured representations of concepts are not tied to a particular architecture, but arise naturally as a consequence of how the network adapts to the probabilities in the data during learning. The information about the magnitude of a number really is in the model, but it is a learned, derived pattern that has to be dug out with a special method; it is not a direct arithmetic operation that the model reliably and always identically taps into when generating an ordinary answer. This unreliability of access to its own knowledge is what produces inaccurate answers when aggregating across dozens or hundreds of values.
The missing data-cleaning phase
What happens inside tokenization and embedding can also be named differently, more practically: through what you yourself do in classic data processing before you calculate anything at all. In Python with pandas, in a SQL database or in Excel there is always an explicit and obligatory data-cleaning phase, which precedes any calculation and any work with data. A deterministic function such as SUM or AVERAGE has no tolerance for inconsistency. If it gets the values „742″, „0742″, „742.00″ or „742 CZK” in one column, they first have to be converted into one unambiguous type, e.g. the float 742.0, otherwise the function fails with an error or returns nonsense.
That is why a data analyst often spends more time cleaning and normalising data than on the analysis itself. From parsing formats, unifying separators (the unfortunate difference between a full stop and a comma in decimal values) and removing currency symbols, to handling missing values. The exact figure differs between surveys, but according to an overview by the company Amperity (2026), which summarises several such surveys, CrowdFlower reported up to 60 % of time spent on cleaning and organising data, while a newer Anaconda survey speaks of roughly 45 % for data preparation overall. All of them agree, however, that data preparation is clearly the single largest time item in a data analyst’s work, considerably larger than modelling or analysis itself. This phase is not optional, it is a precondition for the calculation to be able to run correctly at all.
With a language model, this enforced phase simply does not normally exist. The model does not get the instruction “first unify the format, then calculate” – it just tokenizes the text as it came, with all the inconsistencies inside. Instead of explicit cleaning there is only a learned statistical approximation; the model tries to guess that these notations are similar to each other, but it has no assurance of that, let alone a guarantee like type casting in code. What is a separate, necessary and visible discipline in classic data processing is something the model quietly bypasses with probabilistic guessing.
Attention and the lost-in-the-middle phenomenon
The second big problem appears at the moment when the data is not a few dozen rows but hundreds of them, and they all fit into the context window. You would think that this is a win, since the model has “everything available” at once. The reality is different and it is connected with how the attention mechanism works, attention, which determines which parts of the input the model takes into account when generating its next answer.
In the transformer architecture, every token looks at all the tokens before it during processing. This is called causal masking. It has a side effect: tokens at the start of the sequence are “seen” by absolutely all the tokens that come after them, so more information flow passes through them across the network than through tokens in the middle. Tokens at the end, in turn, benefit from being closest to the place where the model generates its answer. This combination creates a tendency that researchers call “lost in the middle”. The model tends to pay more reliable attention to the beginning and the end of the input, while the middle part tends to be processed worse.
This phenomenon was systematically documented by Liu et al. (2024) from a team at Stanford and Berkeley, who showed on long-context retrieval tasks that the model’s accuracy is highest when the relevant information is at the start or at the end of the input, and drops when it is in the middle. Actually rather similarly to how a human perceives things – the words at the beginning of a lecture and at the end stick in your head more than those in the middle.
It is fair to add, though, that this is a tendency, not an immutable property: newer replication studies from 2026 show that the strength of the effect differs quite a lot between models and datasets and cannot always be reproduced equally easily, so in a specific model and a specific task it may turn out weaker or stronger than the original study suggests.
For practical use it nevertheless still means that when you upload a table of a hundred rows into a model and ask for a sum, an average or the identification of an outlier, the model will probably notice the first and last few rows more reliably, while row forty-seven, where the key anomaly is hiding, runs a somewhat higher risk of falling through the net. Not because the model is stupid, but because the architecture tends towards this behaviour, albeit to differing degrees depending on the specific model.
Hallucination and false certainty
The third and most insidious problem is how the model presents the result. Unlike a SQL query or an Excel formula, where you spot an error from an error message or an obviously nonsensical number, the model produces its answer in the same confident, fluent and structured tone regardless of whether the number really corresponds to reality or the model simply made it up. There is nothing like a confidence score, no ” I’m not sure about this one”, no error message. An estimate and a calculated value look exactly the same from the outside. Hallucination sounds the same as the correct answer.
An explanation of why this happens was given by the work of Kalai et al. (2025) from OpenAI. According to them, models do not guess out of a lack of intelligence, but because standard training and evaluation procedures reward confident answers and punish admissions of uncertainty, so it statistically pays the model to learn to guess with conviction instead of saying “I don’t know”.
From the point of view of results this is more dangerous than a classic error, because a classic error can be checked and traced. You see the formula, you see step by step how the number was arrived at. With a language model no such traceability exists.
Non-determinism, or a different answer every time
One more property is connected with this confidence, and it is often forgotten. Text generation is based on probabilistic sampling of the next token, not on deterministic calculation. That means that the same query over exactly the same data can give a slightly different result on two different runs. For anything that is supposed to be repeatable and auditable (a monthly report, a tax calculation, a comparison of year-on-year growth) this is a fundamental problem. A deterministic function returns an identical result every time. A language model can give you two different numbers twice in a row and both will sound equally convincing.
In its API documentation Anthropic admits that even with the temperature parameter set to 0 the results will not be entirely deterministic, and OpenAI describes it similarly. An independent benchmark by the QAnswer platform (2026) measured this unreliability directly: repeatedly submitting the same longer prompt at temperature 0 led to an identical answer in only a fraction of the runs for some models, because factors beyond the sampling itself come into play, such as floating-point rounding or the way requests are batched on the server.
The illusion of understanding
The last part of the problem is psychological rather than technical. Because the model’s output is fluent, grammatically correct and structured like an answer from a competent analyst, the reader automatically assumes that the model really did “go through” all the data and calculate it. In the user’s head the fluency of the text is mistaken for the correctness of the calculation, which is exactly the type of cognitive shortcut studied by behavioural economics. Remember your school-leaving exams? What did your classmates advise? Talk, talk, talk. If you stumble, everyone can tell you do not know.
Researchers have already named this property too, and they call it processing fluency. Alter and Oppenheimer (2009) argue that the more easily we process a piece of information, the more true and trustworthy it subjectively seems to us, regardless of its actual content. A language model produces text that, by the very nature of being trained on human-written texts, is maximally fluent and coherent, so it triggers precisely this mechanism in the reader within the first few sentences of the answer. We judge the quality of content by the form in which it is presented, not by the process that actually stands behind it.
The Czech number format as an accuracy risk
I have already touched on this lightly, but the Czech environment also plays a role. Models are trained predominantly on English-language data – for the first public models such as GPT-3, English made up over 92 % of the training texts according to the original study (Brown et al., 2020) and for Llama 2 just under 90 % according to the technical report (Touvron et al., 2023) . And we know how it is in English: decimal places are separated by a full stop and thousands by a comma, i.e. “1,234.56”. The Czech notation is exactly the opposite: a decimal comma and a full stop or a space as the thousands separator, i.e. “1.234,56” or with a space and a comma, “1 234,56”.
If the model confuses the conventions, it may accidentally shift the decimal point by three orders of magnitude, and on an invoice or an accounting statement that is not a cosmetic deviation but a difference of an order of magnitude.
The solution? The model as an orchestrator, not a calculator
From everything described above follows one practical conclusion, which I promised at the beginning. And it is actually good news at the same time. The model does not have to calculate itself in order to be useful. It is enough to stop treating it as a calculator and give it the role of an orchestrator. That means that instead of trying to arrive at a number by estimating from tokens, it generates code or a query, which is then executed by a real, deterministic tool – Python, SQL, an Excel formula – and only the result of that tool is returned to the user.
This approach even has a name and solid research backing. In machine learning it is called Program-Aided Language Models (PAL) and it was introduced by Gao et al. (2023). They showed that when the model is restricted to what it is good at (breaking a task into steps and writing code) and leaves the calculation itself to the Python interpreter, accuracy on word maths problems significantly exceeds the approach where the model calculates “in its head” step by step in text.
In practice the difference between these two approaches is a matter of one sentence in the prompt, but the result is fundamentally different. The bad approach looks like this:
„Here is a table of revenue for the last twelve months, calculate the year-on-year growth for me.”
In this case the model will try to answer directly from the context, that is by estimating from tokens, with all the risks described above. The good approach looks like this:
„Here is a table of revenue for the last twelve months. Write and run Python code that calculates the year-on-year growth and show me the result together with the intermediate steps.”
This simple change moves the calculation itself from “guessing on the basis of a language pattern” to real, repeatable and traceable code that gives the same result every time and every step of which can be checked. The model thus becomes what it is genuinely good at and leaves the calculation itself to a tool that is built for it.
Most people do not make this distinction, because the chat interface looks the same whether the model answers off the top of its head or generates and runs code. Learning to recognise when you need to say explicitly “do not calculate it yourself, write me code for it” is probably the most important skill you can take away from this article if you work with data regularly.
A language model is not a bad tool, it is just the wrong tool for calculating straight in its head. It is excellent at understanding context, formulating conclusions, writing code and explaining, but a number it gives you without the help of an external tool is always only an estimate delivered in a confident tone. Once you understand this, it stops limiting you and starts working as an instruction: do not calculate with the model, calculate through it.
References
Alter, A. L., & Oppenheimer, D. M. (2009). Uniting the tribes of fluency to form a metacognitive nation. Personality and Social Psychology Review, 13(3), 219–235. https://doi.org/10.1177/1088868309341564
Amperity. (2026, March 24). Your data scientists were hired to build models. They’re cleaning spreadsheets instead. https://amperity.com/blog/cost-of-poor-data-quality
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. arXiv. https://arxiv.org/abs/2005.14165
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G. (2023). PAL: Program-aided language models. Proceedings of the 40th International Conference on Machine Learning, 202, 10764–10799. https://proceedings.mlr.press/v202/gao23f.html
Jiang, Y., Rajendran, G., Ravikumar, P., Aragam, B., & Veitch, V. (2024). On the origins of linear representations in large language models. arXiv. https://arxiv.org/abs/2403.03867
Kadlčík, M., Štefánik, M., Mickus, T., Spiegel, M., & Kuchař, J. (2025). Pre-trained language models learn remarkably accurate representations of numbers. arXiv. https://arxiv.org/abs/2506.08966
Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). Why language models hallucinate. arXiv. https://arxiv.org/abs/2509.04664
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12. https://aclanthology.org/2024.tacl-1.9/
QAnswer. (2026, April 3). Why LLMs are not deterministic even at temperature 0. https://www.qanswer.ai/blog/llm-non-determinism-temperature-zero
Singh, A. K., & Strouse, D. J. (2024). Tokenization counts: The impact of tokenization on arithmetic in frontier LLMs. arXiv. https://arxiv.org/abs/2402.14903
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv. https://arxiv.org/abs/2307.09288
Unstract. (2025, October 8). Why is deterministic output from LLMs nearly impossible? https://unstract.com/blog/understanding-why-deterministic-output-from-llms-is-nearly-impossible/
