What Does a Model Actually Learn?

No engineer at any AI lab wrote the code that lets a language model conjugate French verbs, name the capital of Australia, close a Python loop or work out that a character who left the room will look for her keys where she left them. The model was given one task, repeated trillions of times: read some text and guess what comes next. Everything else came out of that.

Put that baldly and it sounds like either magic or a trick. Read it as magic and you trust the model with things it does unreliably. Read it as a trick and you miss what it does well, and why.

Here is the reading that holds up. To guess text well, a model is pushed to build internal structure for whatever makes text predictable, and that structure is what we experience as abilities. It's real: researchers can see some of it and even edit it. But it's only as deep as the training text demanded, so it's solid where text exercised it constantly, thin where it didn't, and mixed with shortcuts that work on familiar forms and fail on unfamiliar ones. That one fact explains why a model can solve a hard problem and then fail an easy variant of it, and it tells you how to test whether it will work for you.

The only lesson

A language model reads text as tokens, pieces of words from a fixed menu (The Model Never Sees Your Words). During pretraining, it sees a stretch of tokens and gives a probability to every possible next one. Then it's shown the token that actually came next and charged a penalty: the less probability it gave the real answer, the bigger the charge. Near-certainty in the right answer costs almost nothing. Confidently backing the wrong one costs a lot. Training adjusts the model so the average charge goes down. That's the entire objective, called next-token prediction.

Watch what that simple objective demands. Imagine a training set of seven sentences:

1. The capital of France is Paris.
2. The capital of Italy is Rome.
3. The capital of Spain is Madrid.
4. The weather in Paris is mild.
5. The museum is closed.
6. The answer is yes.
7. The city is beautiful.

The simplest predictor looks at the previous word and counts what followed it. Here, "is" is followed by seven different words. Asked to continue "The capital of France is", the counter gives "Paris" one chance in seven. It can't do better, because the word that decides the answer, "France", sits outside the one-word window it looks at.

A second predictor stores whole sentences. It gets "The capital of France is" right with certainty. Now ask it to continue "France's capital city is". It has never seen that sequence, so it has nothing. A table of whole sentences is perfect on what it stored and useless one rephrasing away.

To lower the penalty on text it hasn't seen word for word, a predictor needs something in between: a way to register that "France" and "capital" both appeared, that they relate, and that the relation points to a particular city. Real models do this with learned internal descriptions of words and their relations. Those descriptions are where the learning lives.

Why predicting well requires knowing things

Scale the seven sentences up to a large share of everything people have written and the same logic applies to every regularity in it.

Facts. A model that has stored "Paris is France's capital", retrievable from any phrasing, pays almost nothing every time the pattern appears. Stored facts are among the cheapest ways to cut the penalty.

Arithmetic. "2 + 2 =" can be memorized. "4,817 + 2,396 =" mostly can't: there are 81 million ordered pairs of four-digit numbers, and the training text contains a small fraction of them. But sums appear constantly, on receipts, spreadsheets, homework and program output. Past a certain range, a procedure beats a table.

Other people's minds. "Sally puts her marble in the basket and leaves. Anne moves it to the box. When Sally comes back, she looks for it in the…" The text continues "basket" because Sally didn't see the move. Predicting that word requires keeping track of who saw what, a small model of someone else's belief.

Text is written about the world, so predicting text well rewards modeling the world it describes.

That's the core of the argument, and it needs a qualifier straight away. The pressure rewards modeling the world only to the extent that it lowers the penalty on the training data. If a shallow shortcut predicts a pattern almost as well as a deep understanding of it, training barely tells them apart, and the model may settle for the shortcut. Everything interesting about these models lives in that qualifier.

Too much text to memorize

The adjustment itself is called gradient descent. A model is a very large set of numbers, its parameters. For each batch of training text, the training software works out which direction a tiny change to each parameter would lower the penalty, and moves every one of them a small step that way. Then it takes the next batch.

Meta's largest Llama 3 model, documented in detail in July 2024, has 405 billion parameters and was trained on 15.6 trillion tokens in about 1.15 million of those steps. Nobody chose any of the final values. The process never searches for "grammar" or "arithmetic". It moves toward whatever lowers the penalty right now, and structure appears because structure happens to be downhill.

There's also a reason the structure generalizes instead of being a giant lookup table: the model doesn't have room for the table. A 2025 study estimated that models of this kind can store about 3.6 bits of information per parameter. If that rate held at Llama 3's size, an extrapolation the study didn't test, the model could hold roughly 180 gigabytes, against training text of roughly 60 trillion characters. Most exact wording can't survive. What survives is whatever serves many documents at once: vocabulary, grammar, frequent facts, common procedures. The same study watched small models memorize examples until their capacity filled up, and then start to generalize instead. Given enough data, general rules win the competition for room, because each bit of a rule serves more text.

Scroll sideways to see the whole figure

What researchers find inside

That a model must build internal structure is an argument. In the last few years researchers have been able to check it. The main tool is a probe: record the model's internal activity while it reads, then train a small separate classifier to read some property of the world off those numbers. If you can then nudge that internal activity and the model's behavior changes the way the property should, the model is actually using it.

The cleanest case is a game. In 2022, researchers trained a small model on nothing but transcripts of Othello games, sequences of moves like "D3 C5 F6", with no board, no rules and no pictures. It learned to propose legal moves almost perfectly, including on games it had never seen and with a quarter of the possible games removed from training, so it wasn't replaying memorized sequences. Then they probed it, and found the board: which tile held which piece could be read off the model's activity. When they edited that activity to flip one tile, the model's predicted moves changed to match the edited board. It had built a board nobody showed it, and was consulting it. A later study added a twist: the model tracks tiles as "mine" or "theirs" relative to whoever moves next, not as black and white. Ask in the model's own terms and the board is plain to see.

Anthropic's researchers have done similar work on production models. Tracing what happens inside Claude 3.5 Haiku in 2025, they found three things.

It plans ahead. Asked for a rhyming couplet starting "He saw a carrot and had to grab it", the model wrote "His hunger was like a starving rabbit". Before writing any of the second line, internal signals for candidate rhymes such as "rabbit" and "habit" were already active. Suppressing "rabbit" produced a line ending in "habit". A model trained to predict one token at a time had learned to pick a destination and write toward it.

It chains facts, and sometimes shortcuts. For "the capital of the state containing Dallas is", there was an internal step through "Texas" before "Austin", and swapping in California produced "Sacramento". A direct Dallas-to-Austin shortcut ran alongside the two-step path.

Its method is its own. Adding 36 and 59, the model ran two paths in parallel, one estimating the rough size of the sum and one computing the last digit exactly. Asked how it got 95, it described carrying the one, the method taught in school. In the researchers' reading, it learned to explain arithmetic by imitating how people explain it, and learned to do arithmetic some other way.

The structure is real. It is also partial, and it is not always what the model says it is.

The researchers are candid about the limits: they got satisfying insight for about a quarter of the prompts they tried, and the published examples are the successes. As of October 2026, nobody has anything close to a complete map of a frontier model.

Real, partial and uneven

So models build genuine representations. The question that matters in practice is how deep they go, and the answer is: as deep as the training text demanded, and no deeper.

The clearest evidence comes from tasks with identical logic and different familiarity. GPT-4 applied "multiply by 9/5 and add 32" correctly, because that's the Celsius-to-Fahrenheit conversion, and got "multiply by 7/5 and add 31" wrong on the same input. It decoded a letter-shift cipher with a shift of 13, a common way to hide spoilers online, far better than the same cipher with a shift of 2. A later test on OpenAI's o1, trained to reason at length, found large improvements and the same direction of effect.

Knowledge can even be stored in one direction. GPT-4 named a celebrity's parent 79 percent of the time, but asked for the child of that same parent, it succeeded 33 percent of the time. Models trained on "A is B" don't automatically learn "B is A". Yet when "A is B" appeared in the prompt, they reversed it without trouble.

And a model can perform beautifully without the structure you'd assume. Researchers trained a model on turn-by-turn taxi routes in Manhattan. It produced valid routes nearly every time, yet the street map implied by its predictions was incoherent, with streets in impossible orientations. Forced into a detour at 1 percent of its steps, it produced valid routes 69 percent of the time; at 10 percent, 8 percent. Good predictions on familiar trips didn't require a map, and without one, detours broke it.

Scroll sideways to see the whole figure

That points to the one test that separates the two kinds of learning: keep the structure of a task and change its surface. A model that learned the rule carries it across. A model that learned the familiar form drops.

Accuracy on the usual cases doesn't reveal which kind of learning produced it. Changing the surface does.

The same logic deflates some of the drama around "emergent abilities", skills that seem to switch on suddenly as models grow. Much of that suddenness comes from all-or-nothing grading. If a model's chance of getting each digit right rises steadily from 50 to 90 to 99 percent, its score on ten-digit answers graded all-or-nothing goes from about 0.1 percent to 35 to 90: nothing, nothing, then a cliff. Measured with partial credit, the same models improve smoothly.

Does it understand?

The argument has hardened into two slogans. One says a language model is a "stochastic parrot", stitching together forms by how they combine statistically, without reference to meaning. The other says it builds a model of the world. Each holds something true.

The evidence above rules out the strong version of each, read as a claim about what's inside. Models build representations that track the world and use them causally, which pure surface statistics wouldn't do. Those representations are often partial, inconsistent and tied to how often a form appeared in training, which a coherent world model wouldn't be. A 2025 study gave the sharpest image: a model trained on planetary orbits predicted trajectories well, even for solar systems it hadn't seen, but the force law implied by its predictions bore no resemblance to Newton's gravity. It had learned to predict like Kepler without arriving at Newton.

Whether that counts as "understanding" is partly a question of definition, and no probe will settle it. My advice is to stop asking it in general and ask three questions that can be answered for a given model and task. Does it track the relevant state? Does that state drive the answer? Does the answer hold when the surface changes?

What this means when you use one

Brilliance and brittleness come from the same mechanism. A model that generalized and one that learned the common form look identical on the common form, so an impressive demo is compatible with either.

Your business lives in the rare forms. Models do better when the task, input and output resemble common text. Internal codes, house abbreviations and a pricing rule nobody outside your company wrote down are exactly the thin territory. When exactness matters, ask for a standard representation, have the model write code for calculations, and spell out unusual conventions.

Put the facts in the context. A model uses what it reads in its prompt more flexibly than what it absorbed in training, as the reversal results show. For anything that must be right about your business, supply the source and ask the model to work from it.

Treat fluency and self-explanation as predictions. The objective rewards plausible continuations, so a confident tone is evidence of nothing (Confidently Wrong). And as the 36 + 59 case shows, a model's account of how it reached an answer is generated from patterns in human explanations. Treat it as a hypothesis to check.

Test on your task, and change the surface. Public benchmarks may be partly memorized and reflect someone else's mix of cases. Take real cases from your work, then vary them: reword the request, reorder the information, use your real formats, include the strange exceptions (Build an Evaluation From Your Own Work).

One consequence deserves more attention than it gets. If models keep improving fastest on common cases, their remaining errors will concentrate in rare ones, and rare errors are the ones people stop checking for because weeks pass without one. A team that relaxed its review because the model has been right for a month now has review tuned to the cases the model already handles. That decision deserves a deliberate second look.

Check yourself

Why does a model that only predicts the next token end up with structure like an Othello board?

Predicting text well rewards modeling what the text describes. Researchers read the board off the model's activity, and editing it changed the predicted moves.

Why can a model solve a hard problem and fail an easy variant?

Its structure is only as deep as the training text demanded. Familiar forms, like the Celsius conversion or a shift of 13, are solid, while the same logic in an unfamiliar form drops.

Why is a model's account of how it got an answer a hypothesis to check?

It is generated from patterns in how humans explain things. In the 36 plus 59 case the model described carrying the one while computing the sum another way.