How language models read text: tokens, byte pair encoding, and why Persian costs more
What a token is, how byte pair encoding builds a vocabulary, why Persian and Turkish take more tokens than English, and how to count tokens instead of guessing.
Parsa Alizadeh4 min read

On this page
A language model never sees letters or words. It sees tokens: numbered pieces of text, often a word, often only part of one. Every limit and every price in an LLM product is counted in them: the context window, the output cap, the bill. This post explains what a token is, how the most common method builds its vocabulary, and why the same sentence is more expensive in Persian or Turkish than in English.
Why not words or characters#
Splitting on words gives short sequences, but the vocabulary never ends. Names, typos, code and new slang all fall outside it, and a model cannot read a word it has no entry for. Splitting on characters has the opposite problem: nothing is ever unknown, but sequences become very long and the model has to learn spelling before meaning.
Subword tokenization sits in between. Frequent words become single tokens, rare words are spelled out from smaller pieces. Sennrich, Haddow and Birch introduced this for translation models: rare and unknown words are encoded as sequences of subword units, using a segmentation based on the byte pair encoding compression algorithm [1].
How byte pair encoding learns a vocabulary#
Byte pair encoding (BPE) starts with the smallest units and repeats one step: count every adjacent pair, merge the most frequent pair into a new unit. The list of merges, in order, is the tokenizer.
A toy corpus of four words with their counts shows it:
low × 5 lower × 2 newest × 6 widest × 3
start l o w l o w e r n e w e s t w i d e s t
merge 1 e + s → es (9 times: newest 6 + widest 3)
merge 2 es + t → est (9 times)
merge 3 l + o → lo (7 times: low 5 + lower 2)
merge 4 lo + w → low (7 times)
result low low e r n e w est w i d estAfter four merges, "low" is one token and "est" is a reusable ending. A word the tokenizer has never seen, such as "lowest", comes out as low + est instead of failing. Real tokenizers run tens of thousands of merges over a very large corpus, but the loop is the same.
The tiktoken README lists the properties that make this useful: it is reversible and lossless, so tokens convert back into the original text, and it works on arbitrary text, even text that was not in the tokenizer's training data. It also gives the practical average: each token corresponds to about 4 bytes [2].
Why the same sentence costs more outside English#
Two things work against languages like Persian and Turkish.
The first is bytes. A tokenizer that starts from bytes, as byte-level BPE does, sees UTF-8. In UTF-8, characters up to U+007F take one byte and characters from U+0080 to U+07FF take two [6]. Every letter of the Arabic script used for Persian, and Turkish letters such as ş, ğ and ı, sit in that second range:
"salam" 5 characters 5 bytes
"سلام" 4 characters 8 bytesThe second is training data. Merges are learned from frequency. A tokenizer trained mostly on English text learns long English pieces and only short pieces for everything else, so other languages are left closer to the raw bytes.
This has been measured. Petrov and colleagues found that the same text translated into different languages can have drastically different tokenization lengths, with differences of up to 15 times in some cases. They point out the consequences: the cost of commercial language services, the processing time and latency, and the amount of content that fits in the context [5].
OpenAI's own guidance is consistent with that. It gives rules of thumb for English (one token is approximately 4 characters, or three-quarters of a word) and adds that these are estimates and that other languages can have different relationships between characters, words and tokens [3].
Other tokenizers#
BPE is not the only method. SentencePiece trains subword models directly from raw sentences, while earlier tools assumed the text was already split into words [4]. That matters for languages that do not put spaces between words. Each model family ships its own tokenizer, so a token count from one is not valid for another.
What to do in practice#
Count, do not estimate. Run the tokenizer that belongs to the model you are calling:
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("Your prompt goes here")
print(len(tokens))A few habits follow from the sections above:
- Measure prompts in the language your users write in. A budget tested only in English will be too small for Persian or Turkish.
- Size retrieval chunks in tokens, not characters, so they mean the same thing in every language. I covered chunking in Retrieval-augmented generation.
- Leave room for the answer. Input and output share the same context window, described in the Transformer, from the original paper.
- Recount when you change models. A different tokenizer gives a different number for the same text.
Summary#
- A token is a piece of text from a fixed vocabulary; limits and prices are counted in tokens.
- BPE builds that vocabulary by repeatedly merging the most frequent adjacent pair [1].
- It is reversible and works on any text, at about 4 bytes per token on average [2].
- Non-English text usually needs more tokens: more bytes per letter in UTF-8 [6] and fewer learned merges. Differences of up to 15 times have been measured [5].
- Rules of thumb are for English [3]. Count with the real tokenizer.
Sources#
- Sennrich, R., Haddow, B., Birch, A. “Neural Machine Translation of Rare Words with Subword Units.” 2015, accepted at ACL 2016. https://arxiv.org/abs/1508.07909
- OpenAI. tiktoken, README (“What is BPE anyway?”). https://github.com/openai/tiktoken
- OpenAI Help Center. “What are tokens and how to count them?” https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
- Kudo, T., Richardson, J. “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing.” 2018. https://arxiv.org/abs/1808.06226
- Petrov, A., La Malfa, E., Torr, P. H. S., Bibi, A. “Language Model Tokenizers Introduce Unfairness Between Languages.” NeurIPS 2023. https://arxiv.org/abs/2305.15425
- Yergeau, F. “UTF-8, a transformation format of ISO 10646.” RFC 3629, section 3. https://www.rfc-editor.org/rfc/rfc3629
- #ai
- #llm
- #tokenization
- #bpe
- #research

Written by Parsa Alizadeh
I build web and mobile products end to end, and the AI tools that make them smarter. I care about fast interfaces, clean systems and shipping.