Token (language model)
In short: What is Token (language model)?
A token is the unit of text a language model processes (a word, word fragment or punctuation mark) in which input and output are measured.
Explanation
A language model processes text neither letter by letter nor strictly word by word, but broken into tokens. A common short word may be a single token, while a longer or rarer word splits into several pieces; agglutinative languages such as Hungarian typically require more tokens per sentence.
Tokens matter in practice for two reasons: providers bill by token count, and they define the context window (how much text the model can keep in view at once) in tokens.
Example: if a long contract exceeds the context window, the system splits it into chunks or passes only the sections relevant to the user's question.