Tokenization

Tokenization is the process of splitting text into tokens according to a vocabulary established during model training. Common words get their own token, rare ones are assembled from smaller pieces - that's why unusual names or words with typos break into many tokens. The tokenization method explains some model quirks: trouble counting letters in words, sensitivity to minor spelling changes, or cost differences between languages. The practical takeaway for users: concise, clean text means fewer tokens, so lower cost and more room in the context.