Token
Definition
The small chunk of text a language model reads and writes, often a word, part of a word, or a punctuation mark.
In plain English
Models don’t see letters or whole words. A tokenizer first cuts text into tokens from a fixed vocabulary, and the model works with those pieces.
How text splits depends on the tokenizer and the language. Languages that are less common in training data, such as Khmer, often need more tokens for the same sentence.
Example
The word “unbelievable” might be split into three tokens such as “un”, “believ”, and “able”. Model limits and prices are usually counted in tokens.
Draft, not yet reviewed by a person · Updated Nov 2026