{"id":9496,"date":"2026-07-27T12:30:29","date_gmt":"2026-07-27T10:30:29","guid":{"rendered":"https:\/\/www.kubicek.ai\/?post_type=lexicon&p=9496"},"modified":"2026-07-28T10:00:24","modified_gmt":"2026-07-28T08:00:24","slug":"tokenization","status":"publish","type":"lexicon","link":"https:\/\/www.kubicek.ai\/en\/lexicon\/tokenization\/","title":{"rendered":"Tokenization"},"content":{"rendered":"

Tokenization<\/strong> is the process of converting raw text into a sequence of discrete units \u2013 tokens<\/a> \u2013 that make up the input alphabet of a language model. Modern LLMs<\/a> work neither with whole words nor with individual characters, but with subword units derived statistically from the training corpus. The most widely used algorithm is Byte Pair Encoding (BPE), which starts at the level of individual bytes and iteratively merges the most frequent adjacent pairs into longer units until the vocabulary reaches its target size (typically 32,000 to 200,000 entries). The alternatives are WordPiece and the Unigram model implemented in SentencePiece. The resulting vocabulary is a trade-off between sequence length and the ability to encode any string at all, including one the model has never seen. Every token is assigned an integer ID, which the first layer of the network turns into an embedding<\/a>. How efficient tokenization is depends heavily on the language: highly inflected languages written with accented characters \u2013 Czech, Turkish or Finnish, for instance \u2013 break down into roughly twice as many tokens as English text carrying the same meaning, which directly drives up both the cost of inference and the consumption of the context window<\/a>.<\/p>\n

\n

Imagine building text not out of letters, but out of pre-printed tiles, as in Scrabble. Some tiles carry a whole everyday word (“house”), others only a fragment (“un-“, “-ing”, “-ness”). The model owns a box with a fixed set of tile types and has to assemble any sentence from those alone. A frequent word takes a single tile; an unusual surname or a typo has to be pieced together from three or four fragments. That is also why the very same passage costs more in one language than in another: “the” is one tile, while an accented, heavily suffixed word may swallow a whole handful. The model pays neither for words nor for characters \u2013 it pays for tiles.<\/p>\n","protected":false},"featured_media":0,"template":"","class_list":["post-9496","lexicon","type-lexicon","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/lexicon\/9496","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/lexicon"}],"about":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/types\/lexicon"}],"wp:attachment":[{"href":"https:\/\/www.kubicek.ai\/en\/wp-json\/wp\/v2\/media?parent=9496"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}