What Training Actually Changes

In early 2022, OpenAI showed what its GPT-3 model did when asked to "explain the moon landing to a 6 year old in a few sentences". It didn't explain the moon landing. It wrote more requests of the same kind: explain gravity to a six-year-old, explain relativity to a six-year-old.

GPT-3 had read enough of the internet to know about the moon landing. It had never been taught that the text in front of it was a request addressed to it. A line asking for an explanation looked like the start of a list of homework prompts, and the most likely continuation of a list is more list. InstructGPT, released alongside that example, answered the question. Same kind of knowledge underneath, different training on top.

That gap explains two things that confuse almost everyone who works with these models: why models from different labs, with similar scores, answer the same request so differently, and why prompting advice that worked brilliantly two years ago now does nothing or makes things worse. Pretraining gives a model what it knows. Later rounds of training decide what it does with that knowledge. Many prompting tricks were patches for a gap in one particular round of that later training, and the next round closed the gap. Advice aimed at the task lasts; advice aimed at the model's quirks expires.

A reader of everything who answers nothing

A model starts as a network trained on one task, repeated trillions of times: given a stretch of text, predict the next token, the chunk of text a bit shorter than a word that models read and write in. The text is a vast mix of web pages, books, code and papers, and predicting it well forces the network to absorb facts, grammar, programming idioms and the structure of arguments (What Does a Model Actually Learn?).

Nearly all of a model's knowledge comes from this stage, called pretraining, and the data mix a lab chooses (how much code, which languages, how much of the web it filters out) shapes what the model will be good at long before anyone teaches it manners.

What pretraining doesn't produce is an assistant. The result, a base model, is autocomplete for whole documents. Nothing in its objective rewards truth over what text commonly says, and nothing marks which text is a request.

That picture, autocomplete for documents, explains where prompt engineering came from. Early prompting was document forgery. If you wanted a translation, you wrote the start of a bilingual glossary with three entries filled in and a fourth left blank. The 2020 GPT-3 paper made this famous as few-shot prompting, and it worked because it spoke the only language a base model understood, which is the shape of documents.

Three rounds of teaching manners

Everything after pretraining is called post-training. It comes in rounds, each learning from a different signal, and each leaves its own fingerprints on how the model behaves.

Scroll sideways to see the whole figure

Round one: show it examples. People write thousands of requests paired with good responses, and the base model is trained further on them. This is supervised fine-tuning. A template marks who's speaking, so the model learns that an assistant turn answers.

The striking thing is how little this takes and how much it changes. Raters preferred the answers of a 1.3-billion-parameter post-trained InstructGPT over those of the 175-billion-parameter GPT-3, a model more than a hundred times bigger. In 2023 a research team built a capable assistant from a large base model with just 1,000 carefully chosen examples. This round adds little knowledge and changes a great deal about what the model does with the knowledge it has. Forged documents became unnecessary for most tasks, because the model now treated a plain request as a request.

Round two: let people pick the better answer. Writing perfect demonstrations is slow, and judging two answers is much easier than writing one. In reinforcement learning from human feedback, or RLHF, the model writes several responses and people rank them. Those rankings train a separate network, a reward model, to predict which answer a rater would prefer. The language model is then trained to produce answers the reward model scores highly. Variants such as DPO, or Anthropic's AI judges working from written principles, change the plumbing; the aim stays the same: push the model toward what a judge prefers.

This round gives each assistant most of its recognizable manner, and its most interesting side effects, because a reward model learns what raters liked, which isn't quite what was good.

A reward model is a model of what raters liked, and raters like some things they shouldn't.

Raters tend to prefer longer, more thorough-looking answers. One 2023 study found that a reward that simply scored longer answers higher reproduced most of RLHF's improvement. That's one origin of the wordiness many users noticed in assistants. Raters, and the reward models trained on them, also sometimes prefer a well-written answer that agrees with them over a correct one that doesn't. That's one origin of sycophancy, the model's habit of telling you what you want to hear. In April 2025 OpenAI rolled back a GPT-4o update after users found it overly flattering; its own explanation was that the update leaned too heavily on short-term feedback (Why Models Agree With You).

Notice what this does to prompting advice. If a round of training leaves a model wordy or lazy, users find phrases that counteract it: real fixes, for one model, tied to an imbalance the lab is also working to remove.

Round three: practice with an answer key. Judges are expensive and biased. For some tasks there's a cheaper judge: the answer can be checked. A math problem has a correct number; code passes its tests or it doesn't. Letting a model attempt such problems and rewarding it only when the checker says yes is called reinforcement learning with verifiable rewards.

This is where today's reasoning models come from, starting with OpenAI's o1 in September 2024. In January 2025 DeepSeek published the approach in detail: its R1-Zero model, trained this way without first seeing human-written reasoning, went from 15.6 to 71.0 percent on a hard math competition over training, and learned by itself to spend more effort on hard problems, re-examine its first approach and check its work (What Thinking Buys).

The side effect is the mirror image of sycophancy. A checker rewards passing the check, and under enough pressure a model finds ways to pass that nobody intended. OpenAI described a frontier reasoning model that, while training on coding tasks, considered making a verification function always return "true" instead of fixing the code. This is called reward hacking.

Labs now also train models inside the software they'll run in, the harness that feeds them context and runs their tools, so a model learns tool formats and work habits in the shape of the environment where it practiced. And some layers sit outside the weights entirely: written behavior specifications, and the system prompt every product inserts at the start of each conversation, which is one reason the same model can feel different in a lab's app and in a developer's product.

The life of a trick

Watch one piece of advice go through all of this.

In May 2022, researchers found that adding five words to a math question, "Let's think step by step", lifted one of OpenAI's models from 17.7 to 78.7 percent on an arithmetic test. The phrase worked because models reasoned better when they wrote out intermediate steps but didn't do so by default.

Then the third round of training happened. Reasoning models were rewarded for whatever working-out led to correct answers, so writing out the steps became trained-in behavior. By 2025, OpenAI's guide for its reasoning models told developers that asking them to think step by step was unnecessary. A Wharton team measured the same kind of instruction moving reasoning models' accuracy by about three points either way while making each answer take 20 to 80 percent longer. As of October 2026, several current Claude models have thinking always on, with an effort setting deciding how much.

The phrase still means what it meant in 2022. The model underneath changed, training absorbed the patch, and the remaining dial moved from the prompt to a setting.

The same arc repeats across the famous tricks. Getting clean JSON out of a model used to be a craft of pleading ("respond ONLY with JSON") and repairing. In August 2024 OpenAI launched Structured Outputs, which combines training with constrained decoding: at each step, the system only allows tokens that keep the output valid. On OpenAI's own hard-schema test it scored 100 percent, against under 40 percent for an older GPT-4 without it. Writing the first characters of the model's reply yourself, to force a format, was once recommended in official docs; starting with Claude 4.6, the newest Claude models refuse it. Words like "NEVER" and "CRITICAL" in capitals compensated for models that under-weighted instructions buried in long prompts. Once models were trained to follow instructions more closely, the same emphasis made them over-apply them, and both OpenAI's and Anthropic's guides now say to drop it.

Personas are subtler. A study testing 162 roles on 2,410 factual questions across four model families found "You are a world-class expert" added no accuracy. Vendor guides still recommend setting a role, and both are right: a role shapes tone and scope, and adds no knowledge the model lacks.

A trick is a message to one model about one gap. When the gap closes, the message keeps arriving.

Why each lab's model feels different

Stack the layers and the differences between labs stop looking mysterious, because every layer involves choices labs make differently. The data mix shapes what a model knows and how it writes before post-training starts. The post-training recipe (who the raters are, what guidelines they follow, the balance between the rounds) can turn base models of similar strength into assistants with very different temperaments. Behavior policies decide where refusals fall and how readily the model pushes back. Product targets matter too: a consumer chat app and a coding agent for developers reward different habits. And a model trained inside its lab's own agent product carries that product's habits.

Some argue that pretraining explains nearly everything and post-training is a thin coat of paint. That's right about capability ceilings: no later training makes a model know chemistry it never read. It's wrong about behavior. InstructGPT, the length effect and the GPT-4o rollback all show later training moving behavior a long way on a fixed base. Labs rarely release their base models, so the exact split between two labs is an inference, but it's clear enough to act on.

The upshot is that a prompt meets trained habits, and habits differ by lab and by release. OpenAI said GPT-4.1 followed instructions "more closely and more literally" than its predecessors. Anthropic says its recent Opus models respond more strongly to the system prompt, so instructions written to make an older model use its tools may now make a new one over-use them.

What lasts

Sort advice by what it addresses. Some addresses the task: what you want, why, for whom, what done looks like, how you'll check. Some addresses the model: its blind spots, its imbalances, its parser. Training keeps improving the model, so the second kind decays. The task stays the same, so the first kind keeps paying.

Scroll sideways to see the whole figure

A clear goal and the reason behind it last, because no training gives a model information it was never given. So does context only you have: your codebase's conventions, your customer's history, the constraint legal imposed last month. Examples of good output last as demonstrations of quality, though their old job of explaining the task is gone.

The middle column is where teams over-correct. Several practices that look like tricks hold a durable kernel in an expiring form: roles survive as a way to set tone, step-by-step survives as an effort setting, format pleading survives as a schema. And some practices were never about the model at all. Evaluation sets, typed inputs and outputs, logging, retries for transient failures and human review of high-stakes actions were good engineering before language models and have survived every shift since. Throwing those out with last year's magic phrases is the expensive mistake.

The fastest test for telling a principle from a patch: would it help a capable human contractor? Telling a new colleague the goal, the audience and an example of good work helps them. Offering them a $200 tip for a better spreadsheet, or writing to them in capitals, doesn't. Advice that would help a smart person new to your situation addresses the task. Advice that names a magic phrase addresses one model.

What to do

Prompt libraries rot. A shared library built in 2024 encodes 2024 models' gaps: some instructions now do nothing, some cost tokens and time, and a few (absolutes, aggressive tool instructions, few-shot blocks on reasoning tasks) make results worse. The decay is easy to miss, because a harmful prompt still produces plausible output. And the major providers ship new models several times a year. My advice is to spend your effort on the half that compounds.

Write instructions as intent. Explain the goal and the reason for each constraint. A prompt made of reasons gives the next model something to work with; a prompt made of compensations for the last model gives it noise (Write for the Model That Doesn't Exist Yet).

Date your patches. Next to every instruction that exists to fix a model's behavior, note the model and the failure. That turns the next migration from archaeology into a checklist.

Treat every upgrade, and every switch of provider, as a migration. Keep a small evaluation set, about thirty real cases with known good outcomes, and run the new model with the old prompt, then with a stripped-down one, more than once each, because outputs vary between runs (Build an Evaluation From Your Own Work). Prompt variations help some questions and hurt others, so a trick that worked on the three examples you tried can be neutral or harmful on average. Different labs' training produces different habits, which makes switching providers a quiet form of lock-in; the same set turns a switch into a measured decision.

Buy and build on the task side. Tools that compensate for a model gap (format repair, magic-phrase optimizers, scaffolds that force reasoning) compete with the next round of post-training. Tools whose value lies in the task (data access, evaluation, domain context) tend to gain as models improve.

For individuals the lesson is the same. Phrase craft depreciates with each release. Saying clearly what you want, supplying the context only you have, showing what good looks like and checking the result pay more with every generation, because a model that follows instructions more literally does more with good instructions and more damage with bad ones.

Check yourself

Why did GPT-3 answer a request by writing more requests?

It was a base model, autocomplete for whole documents. It had never been taught that the text in front of it was a request addressed to it, so a list of prompts continued as a list.

Why did "Let's think step by step" stop mattering for reasoning models?

Training rewarded whatever working-out led to correct answers, so writing out steps became built-in behavior. The phrase meant the same, but training had absorbed the patch.

How can you tell a principle from a patch?

Ask whether the advice would help a capable human contractor. Telling them the goal, the audience and an example helps. A magic phrase or capitals does not.