The Model Never Sees Your Words

Take a paragraph in English and the same paragraph professionally translated into Spanish. Send both to a model running on OpenAI's current tokenizer, the one behind GPT-4o and GPT-5. The Spanish version needs about a third more of the units the model reads and you pay for. In 2020, on OpenAI's tokenizer of the time, it needed twice as many. On DeepSeek's tokenizer, Chinese needs fewer than English.

Neither language changed between 2020 and today. What changed is a layer most people never look at.

Before a model reads your text, a small, fixed program cuts it into pieces and swaps each piece for a number. The pieces are called tokens, and the program is the tokenizer. Here is the whole argument in one paragraph: the model never sees your words, only numbered pieces from a menu that records what was common in the text its builders collected. That menu decides what the model can notice, how much of your material fits, and how many units you're billed for. Choosing it is a market decision, even when nobody experiences it as one.

What actually arrives

Here is one sentence, cut the way OpenAI's current tokenizer cuts it, in four languages. Each block is one token.

Scroll sideways to see the whole figure

Three things stand out. The leading space belongs to the token, so " meeting" with its space and "meeting" without one are different pieces. Common words arrive whole and rarer ones are stitched together from fragments: "rescheduled" becomes res + cheduled. And the menu plainly has favorites. It has a single piece for the Spanish reunión but not for aplazó, and almost nothing for Basque.

Then the pieces become numbers. In this tokenizer, " contract" is token 6698 and " the" is token 290. Inside the model, each number points to a row in a huge table of learned values, called an embedding, and everything the model knows about "contract" lives in how that row relates to all the others. The model never looks inside the piece. It has no direct way to see that "contract" starts with a c.

Why build it this way instead of feeding the model letters? Because letters make sequences long, and long sequences are expensive. In English, OpenAI's current tokenizer packs about five characters into each token. Reading letters instead would make every input about five times longer. In a standard Transformer, the architecture behind today's models, the attention step compares every position with every other, so that part of the work grows with the square of the length: roughly 24 times more for the same text. Generating would take about five times as many steps too, since the model writes one piece per step.

So a tokenizer is a compression scheme, a trade between a bigger menu and shorter sequences. Byte-level models show it can be removed, but as of October 2026 the frontier models people use every day still read tokens.

Where the menu comes from

Nobody writes the menu by hand. It's learned from a large pile of text with a procedure simple enough to fit in a paragraph, called byte-pair encoding.

Start with single characters. Count which pair of neighbors appears most often across the whole pile. Glue that pair into a new piece and add it to the menu. Count again, glue again. Stop when the menu reaches the size you chose: about 50,000 pieces for GPT-3's tokenizer, 100,000 for GPT-4's, 200,000 for GPT-4o's.

Here it is running on a toy pile of fifteen words.

Scroll sideways to see the whole figure

The procedure only cares about frequency. "The" becomes one piece because it's everywhere. "On" stays as two letters because, in this pile, it appears once. Scale that up to billions of words and the conclusion follows on its own.

A tokenizer is a record of what was common in the text it learned from.

If most of that text is English, English words get whole pieces and everything else gets assembled from fragments.

Blind below the piece

The first consequence is the famous one. Chatbots struggled for years to count the r's in "strawberry".

Mid-sentence, " strawberry" is a single token. Ask "How many r's are in strawberry?" and the model receives eight numbers, one of which stands for the whole fruit. That puts it roughly where you'd be if someone held up 🍓 and asked how many r's it contains. You can answer, because at some point you learned the spelling. But you can't look. You have to remember, then count inside your head.

Models do learn spelling, from all the spelled-out words, rhymes and crosswords on the internet. Counting is a second operation stacked on a remembered spelling, and that stack was wobbly. Research in 2024 found models could identify a word's letters and still miscount the repeated ones, so the counting itself was part of the problem. The tokenizer makes the task harder; exactly how much of the failure it explains is still unsettled. Newer models mostly get strawberry right, partly because the example became famous and partly because reasoning models often write the word out first: s, t, r, a, w… Once the letters exist as separate tokens, the question changes from memory to reading.

Numbers get the same treatment. On OpenAI's tokenizers, 2026 arrives as "202" followed by "6", and 1234567 without commas as "123", "456", "7". Nothing in those chunks says which digit is the units, and the same digits land in different chunks as numbers grow. The best models still learn arithmetic remarkably well, but they learn it around an awkward representation. Meta's first LLaMA models and Alibaba's Qwen avoid it by splitting every number into single digits.

When a task depends on characters, make the characters visible.

The rule is practical. Character limits, rhymes, anagrams, letter-level edits, word counts, checking that a code has exactly eight digits, exact arithmetic: all of it lives below the level the model sees. Ask it to spell the thing out, or better, to write and run a few lines of code. A model is unreliable at seeing letters and good at writing a program that does.

The language premium is partly a choice

The second consequence is the one that reaches the bill.

Some of the Spanish premium belongs to the language. In the same professionally translated sentences, Spanish uses about 19 percent more characters than English to say the same thing. No tokenizer is obliged to hide that.

The rest belongs to the tokenizer, and you can watch it move by training small tokenizers yourself with the same algorithm and menu size but different diets. Fed 90 percent English, a tokenizer needs 1.46 times as many tokens for the Spanish version. Fed half and half, 1.15, close to the character gap. Fed 90 percent Spanish, the Spanish text needs fewer tokens than the English. Same sentences; only the diet changed.

At scale the picture looks like this.

Scroll sideways to see the whole figure

On OpenAI's current tokenizer, the large Western European languages need 20 to 50 percent more tokens than English for identical content: Spanish 1.32 times, German 1.31, French 1.37, Italian 1.47. Smaller languages of the same region tend to need more than their larger neighbors: Catalan 1.48, Basque 1.60. On the Qwen and DeepSeek tokenizers, built by Chinese labs, the big four European languages need 50 to 63 percent more, while DeepSeek's reads Chinese more compactly than English and Qwen's reads Chinese and English at parity.

Writing systems add a handicap before training even starts. These tokenizers begin from bytes, the raw units computers store text in. A Latin letter takes one byte; a Devanagari letter, used for Hindi, takes three. The same Hindi sentence starts out more than twice as long in bytes as its English equivalent, and the learned merges have to climb back from there.

The menu also moves between model generations. OpenAI's 2024 tokenizer made Hindi roughly three times more compact relative to English than its 2023 one, with no change in price per token. It can move the other way. When Anthropic released Claude Opus 4.7 in April 2026, it said the new tokenizer maps the same input to roughly 1.0 to 1.35 times as many tokens depending on the content, and its pricing page puts it at about 30 percent more for the same text. The rate card didn't have to change for the number of billed units per task to move.

What it does to your bill, and what it doesn't

It's tempting to read "Spanish 1.32" straight onto an invoice. That overstates it for most real work.

The ratio applies to prose. Code costs about the same whatever language your team speaks: def calculate_total(items): is five tokens on OpenAI's current and previous tokenizers and on Qwen's and DeepSeek's. Data formats and numbers are close to language-neutral too. Reasoning models add another wrinkle. Their hidden working, the thinking tokens billed as output, often drifts into English even when the question was asked in another language, so the premium may apply there only partly.

A rough rule for the token part alone: the extra cost is about the prose share of your tokens times the ratio minus one. If prose is 20 to 40 percent of what a coding agent reads and writes, working in Spanish adds a few percent to the low teens. For a customer-service assistant, which is nearly all prose, it's about a third. That's real money at scale and rarely the biggest lever. How a task is framed, how much irrelevant material gets pushed into context and how many times an agent loops before finishing routinely move costs by multiples (Where the AI Money Goes).

The exception is languages the menu barely covers. Amharic needs nearly six times as many tokens as English on OpenAI's current tokenizer. At that ratio, the premium alone shapes what a business can afford to build.

The same premium shows up in two other places. Room: a context window that holds 100 pages of English prose holds about 76 of Spanish, and for English, 200,000 tokens is roughly 160,000 words (What Fits Versus What Gets Used). Time: a model writes a roughly steady number of tokens per second whatever the language, so the Spanish version of an answer takes about a third longer to appear.

Does it think worse in Spanish?

That's the harder question, and the honest answer is that two causes are tangled. Fragmenting is itself a handicap: the model builds meaning from more, smaller pieces, each seen less often. Controlled experiments on smaller models, identical data and different tokenizers, found English-centric tokenizers clearly hurt multilingual performance; nobody has run that test at frontier scale. But the languages that get fragmented are usually the ones with less text in the training data, and less data means less learned. From outside you can't separate the two, which is why a token ratio is a poor measure of quality.

For major languages the news is mostly good. On a translated knowledge test, a stronger reasoning model narrowed the gap between Spanish and lower-resource languages like Swahili and Yoruba, though the language with the least data still trailed furthest. And Anthropic, tracing what happens inside its Claude models, found concepts shared across English, French and Chinese, with language-specific machinery concentrated at the input and output, and a larger shared fraction in a bigger model. Its researchers add that English remains, in a meaningful sense, privileged. Understanding looks largely shared, with a tilt toward the language the model saw most.

So for Spanish, French or German, translating prompts into English as a precaution is usually unnecessary; test it on your own task before paying the cost in clarity. There's a better argument for picking one working language for instructions, documentation and code, and it has little to do with intelligence. Mixed-language instructions create small ambiguities (is "cliente" the same field as "customer"?), and ambiguity costs more than tokens.

Who sets the menu sets the unit

Step back and the tokenizer stops looking like a technical detail. It's one of the places where the economics of AI get decided, quietly, before anyone writes a pricing page.

Per-token pricing passes the builder's data mix on to the customer. A provider that trained its tokenizer mostly on English charges every non-English customer more units for the same content, a surcharge that never appears as a line item. Chinese labs ship tokenizers that favor Chinese. OpenAI cut the relative cost of Hindi by about two thirds in one generation. Whoever wants a language market has to fix this layer first.

Two business consequences follow. Cost models built on English case studies are off for everyone else: a Spanish customer-service deployment should expect around a third more tokens on OpenAI's current tokenizer than the English pilot it was budgeted from, and about half again as many on Qwen's or DeepSeek's, while a Spanish-speaking engineering team running coding agents can mostly ignore the issue. And a model upgrade can quietly be a unit change. Teams that track cost per finished task see it immediately. Teams that track price per million tokens see nothing until the invoice arrives.

Some companies now budget tokens per team the way they once budgeted cloud spend. That's a useful unit in the way a kilowatt-hour is useful: it measures what was consumed. Neither tells you what got done, and both look different depending on who built the meter.

What to do

Make characters and digits visible. When a task depends on letters, exact counts or arithmetic, ask the model to spell it out or, better, to use code.

Compare providers on cost per finished task, with your own material. A price per token only means something once you know the token, and each provider defines its own and can redefine it with a release (Cost per Task, Not Cost per Token).

Measure context in your documents, not in tokens. The useful question is how many of your contracts or tickets fit.

Budget non-English work from its own numbers. Expect roughly 20 to 50 percent more tokens for prose in major European languages on OpenAI's current tokenizer, more on others, much more for many Asian and African languages, and about the same for code and data.

Test quality in your language, on your task. Don't infer it from the token ratio.

The broader habit is the one worth keeping. When an AI system does something oddly stupid, ask what it actually received. Very often the answer sits one layer below the intelligence: in how the text was cut, what was left out of the context, or what the surrounding software chose to show it. The tokenizer is the first of those layers. Same Model, Different Results looks at the rest.

Check yourself

Why did chatbots struggle to count the r's in strawberry?

Mid-sentence the word is a single token, so the model gets one number and cannot look at the letters. It has to remember the spelling and count in its head.

Why can a model upgrade raise your bill when the price per token has not changed?

A new tokenizer can turn the same text into more tokens, so the billed units per task move even though the rate card stays the same.

Why does Spanish add only a few percent for a coding agent but about a third for a customer-service assistant?

The extra tokens apply to prose. Code costs about the same in any language, and a customer-service assistant is nearly all prose.