Contact

Token (language model)

Category
Artificial intelligence
In Hungarian
Token (nyelvi modell)
Entry
84 / 90

In short: What is Token (language model)?

A token is the unit of text a language model processes (a word, word fragment or punctuation mark) in which input and output are measured.

Explanation

A language model processes text neither letter by letter nor strictly word by word, but broken into tokens. A common short word may be a single token, while a longer or rarer word splits into several pieces; agglutinative languages such as Hungarian typically require more tokens per sentence.

Tokens matter in practice for two reasons: providers bill by token count, and they define the context window (how much text the model can keep in view at once) in tokens.

Example: if a long contract exceeds the context window, the system splits it into chunks or passes only the sections relevant to the user's question.

All terms from A to Z

Have an idea or an engineering challenge to solve?

Let's talk