AI Tokens: What They Are and Why Non-English Text Costs More
Tokens are the smallest text units AI models process. Learn how they affect API costs, context limits, and why some languages use more tokens than others.
👁 117 reads
A token is the smallest piece of text that a model breaks down what it receives and what it writes. It’s not a letter or a word, but something in between: a common word is often one token, while a rare one breaks into several pieces. It’s the unit used to measure limits, memory, and API bills, so it’s worth understanding how it works.
How Many Characters Is a Token
For English, the rough estimate is that a token equals about four characters, or three-quarters of a word. Heavily inflected languages with diacritics fare worse because inflection and special characters break words into more pieces; the same text in such a language can take noticeably more tokens than in English. Practical takeaway: when billing by tokens, heavily inflected languages are more expensive than English for the same content. You can estimate the order of magnitude by dividing the character count by three; for an exact character count, use a character counter.
Where Tokens Matter
- Context window. A limit of “128 thousand tokens” applies to the entire conversation including documents, not to a single question.
- API pricing. You pay separately for input and output tokens, and output tokens are typically several times more expensive.
- Speed. The model generates token by token, so a longer response means longer wait times.
- Truncation. When you exceed the window, the oldest part of the conversation drops out and the model stops remembering it, often without warning.
How to Save Tokens
Don’t paste entire documents if you only need one chapter. Start a new conversation instead of dragging a thread through a hundred messages, because the entire history is counted every time. For batch tasks, use a smaller model for simple steps and reserve the large one for where it’s truly needed. I collect pricing for individual plans and models in the pricing hub.
Pricing data: Model atlas, where each model lists the price per million tokens.
Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →Frequently Asked Questions
Is a Token the Same as a Word?
No. A word can be one token or several, depending on how frequently it appears in the data used to train the tokenizer.
Why Does Polish Use More Tokens?
Because inflection produces many rare forms, and rare character sequences are broken into smaller pieces. This is a feature of the tokenizer, not malice from the provider. We tested this with Polish, a heavily inflected Slavic language, as a representative example of non-English tokenization costs.
Does Token Count Affect Response Quality?
Indirectly: the closer you get to the window limit, the greater the chance the model will lose track of earlier points. More about the models themselves in the LLM article.
More on this from QUANTSLANT
guides·RAG: How Retrieval-Augmented Generation Works and Where It FailsRAG combines search with AI generation to answer questions from your documents. Here's how it works, when to use it, and where it typically breaks down.