Token

A token is the basic unit a language model works with - a piece of text, usually shorter than a word. In English, one word averages just over one token; other languages with complex grammar and special characters can be "more expensive" and the same sentence may take more tokens. Tokens are the currency of the AI world: they measure context windows, usage limits, and API pricing (cost per million input and output tokens). When a tool says text is too long - it's about the number of tokens, not characters.