A token is the basic, atomic unit of information into which input text is divided during the process called tokenisation. In natural language processing it is a segment of text that the model handles as a single whole. Depending on the tokenisation algorithm used (BPE, WordPiece), a token may be a whole word, a part of one (a subword), or even a single character. The process is essential because neural networks cannot work with text in its raw form; instead they process sequences of tokens, which are then converted into numerical vectors.
Imagine that a computer cannot read whole sentences at once but needs first to break them down into smaller, more digestible pieces, much as when you build something out of Lego. A token is exactly one such piece – most often a single word or a meaningful part of one. The model therefore first carefully chops the whole text into these basic building blocks and only then starts working with them. This approach lets it understand language better and even cope with words it has never seen, because it can assemble them from familiar smaller parts.