ChatGPT has never read a single word. How tokens work

When you type to ChatGPT, it never sees your words. First, your text is cut into small chunks. These chunks are called tokens. Then each chunk is swapped for a number. ChatGPT only ever works with those numbers. It even writes its answer as numbers, one chunk at a time, and they are turned back into text for you.

This one fact explains a lot. It is why AI used to get the r's in "strawberry" wrong. It is why the same sentence costs more in Hindi than in English. And it is why a long chat runs out of room. Below we show what tokens are, how to see them yourself in two minutes, and the real counts we got.

1. What a token is

A token is a chunk of text that shows up a lot. Sometimes it is a whole word. Sometimes it is part of a word, or one letter, or a space plus a word. OpenAI's own rule of thumb: in normal English, one token is about four letters, or about three quarters of a word. So 100 tokens is about 75 words (OpenAI tokenizer).

Here is "strawberry" in the version used by GPT-4o and OpenAI's newer models:

strawberry                       ->  st | raw | berry        ->  302, 1618, 19772
How many r's are in strawberry?  ->  How | many | r | 's | are | in | strawberry | ?
                                     ' strawberry' is ONE token: 101830

On its own, "strawberry" is three chunks. With a space in front, the way it sits in a sentence, it is one chunk. Capital letters change the chunks again: "Strawberry" is Str | aw | berry and "STRAWBERRY" is ST | RAW | B | ERRY.

For the curious: where the chunks come from. The list of chunks is fixed before the model is trained, using a method called byte pair encoding (BPE). Pieces of text that often appear together are merged into one token, again and again, until the list is big enough. The version above is called o200k_base. OpenAI's tiktoken library describes BPE as reversible and lossless, with each token covering about 4 bytes of text on average (tiktoken on GitHub).

2. See the tokens yourself

On OpenAI's tokenizer page

1. Go to platform.openai.com/tokenizer. No login is needed.
2. Pick a tab. "GPT-5.x & O1/3" is the current tokenizer; "GPT-4 & GPT-3.5 (legacy)" is the older one.
3. Type or paste any text. The page shows the token count, the character count, and each token as a coloured block.
4. Click Token IDs to see the numbers the model actually receives.

In five lines of Python

Install OpenAI's tokenizer library with pip install tiktoken, then run:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
ids = enc.encode("How many r's are in strawberry?")
print(len(ids), ids)
print([enc.decode([i]) for i in ids])

Output (tiktoken 0.14.0, run on 18 October 2026):

8 [5299, 1991, 428, 885, 553, 306, 101830, 30]
['How', ' many', ' r', "'s", ' are', ' in', ' strawberry', '?']

Swap "o200k_base" for "cl100k_base" to see how GPT-4 and GPT-3.5 split the same text. These are the numbers the tokenizer page showed us for the same sentence, so the two methods agree.

3. The counts we measured

All counts below come from tiktoken 0.14.0 and were checked against OpenAI's tokenizer page on 18 October 2026. "GPT-4 era" is cl100k_base (GPT-4, GPT-3.5); "GPT-4o and newer" is o200k_base.

The greeting is the one OpenAI used when it launched GPT-4o: "Hello, my name is GPT-4o. I'm a new type of language model, it's nice to meet you!" and its Hindi translation. OpenAI published those counts (English 27 to 24, Hindi 90 to 31) and our run reproduced them exactly (OpenAI, Hello GPT-4o). The other sentences are our own translations, so treat them as one example each, not a benchmark.

4. Why it matters

Cost

API prices are quoted per million tokens, not per word (OpenAI API pricing). When we estimate a bill, we count tokens. A language that needs more tokens for the same meaning costs more to send and receive, and takes longer to generate.

Context limits

A model's context window, the most it can handle in one request, is also measured in tokens. OpenAI's docs define it as "the maximum number of tokens that can be used in a single request", including input, output and reasoning tokens (OpenAI, conversation state). Text that splits into more tokens fills that window sooner.

Spelling, letter counting and maths quirks

If " strawberry" arrives as one number, the letters inside it are not directly visible to the model. It has to have learned from training which letters that number stands for. That is the usual explanation for why models miscounted the r's in strawberry, struggled to spell words backwards and stumbled on some digit-level arithmetic.

Researchers mostly agree tokenization is a big part of it, but not the whole story. One study found counting errors track the number of letters and tokens in a word (Fu and colleagues, 2024). Another tested the popular explanations and found tokenization alone does not account for the failures, and that getting the model to reason before answering was the most robust fix (Xu and Ma, 2024).

That matches what you see today: newer reasoning models usually count the r's correctly, because they write the word out letter by letter before answering. The tokens did not change; the model learned to work around them.

A practical fix: when letters matter (spelling, counting, reversing, codes), ask the model to write the word out with a space between every letter first, or to use code. Then check the answer.

Hindi and other languages

Tokenizers are built mostly from the text they were trained on, and English dominates that text. So other languages often split into more, smaller pieces. A NeurIPS 2023 study found that translations of the same text can differ in token length by up to 15 times across languages, which means some speakers pay more and get less context for the same meaning (Petrov and colleagues, 2023).

OpenAI's newer tokenizer narrowed the gap a lot for Indian languages. In its GPT-4o launch post, the same greeting fell from 90 to 31 tokens in Hindi, 145 to 33 in Gujarati, 159 to 45 in Telugu, 116 to 35 in Tamil and 96 to 33 in Marathi (OpenAI, Hello GPT-4o). Hindi still needs more tokens than English for that greeting (31 against 24), but the difference is now closer to 1.3 times than 3 times.

Limits of this guide

• Counts are for OpenAI's tokenizers only. Claude, Gemini and Llama use their own, and the same text splits differently.
• Which encoding a model uses is taken from tiktoken's model list (it maps GPT-4o, GPT-4.1, o1, o3 and GPT-5 to o200k_base) and from the tokenizer page's tabs.
• The ChatGPT app adds its own instructions and formatting around your message, so the tokens it sends are more than the ones you typed.

Sources

All opened on 18 October 2026.

• OpenAI tokenizer: platform.openai.com/tokenizer
• tiktoken, OpenAI: github.com/openai/tiktoken
• Hello GPT-4o, OpenAI, May 2024 (language tokenization section): openai.com, Hello GPT-4o
• OpenAI API pricing: OpenAI API docs, pricing
• Conversation state and context windows, OpenAI: OpenAI API docs, conversation state
• Language Model Tokenizers Introduce Unfairness Between Languages, Petrov, La Malfa, Torr and Bibi, NeurIPS 2023: arxiv.org/abs/2305.15425
• Why Do Large Language Models (LLMs) Struggle to Count Letters?, Fu, Ferrando, Conde, Arriaga and Reviriego, 2024: arxiv.org/abs/2412.18626
• LLM The Genius Paradox: A Linguistic and Math Expert's Struggle with Simple Word-based Counting Problems, Xu and Ma, 2024: arxiv.org/abs/2410.14166