Think of chunking as cutting a thick book up into smaller cards – each holding exactly one self-contained idea or paragraph. When you then want to find a particular piece of information, you do not leaf through the whole book. You reach straight for the card whose content most closely matches your question. Without this splitting, an AI would have to work through entire documents on every query, would run into the limits of its memory, and the answers would be less accurate.
Chunking is the technique of dividing a longer text document into smaller, semantically coherent parts – chunks. It is a key pre-processing step in the Retrieval-Augmented Generation (RAG) architecture and generally in any system built on vector search. The reason is twofold: language models have a limited context window, so entire large documents cannot be placed in it, and at the same time retrieval accuracy falls if too many different topics are mixed within one indexed unit. A short, thematically clean chunk is found far more reliably in semantic search than a long passage mentioning five things at once.
In practice there are several strategies for splitting a document. The simplest is fixed-size splitting (for example 512 tokens per chunk), which is fast but often cuts an idea off mid-sentence. A sliding window adds overlap between neighbouring chunks so that context is not lost at the boundary. More advanced structural chunking respects the document’s natural structure and splits by headings, paragraphs or lists. The most sophisticated is semantic chunking, which places the boundaries between blocks where the topic changes in the text – not where a numeric limit happens to fall. Each resulting chunk is then turned into a vector by an embedding model and stored in a vector database, where the most similar one is retrieved at query time.