Same Model, Different Results

In April 2026 the security firm Endor Labs ran the same coding tasks through OpenAI's GPT-5.5 twice. Inside Cursor, a third-party coding tool, the model produced working code for 87 percent of them. Inside Codex, OpenAI's own tool, it managed 62 percent.

Same model. Same tasks. A gap of more than 25 points, and nothing changed except the software the model ran inside.

That result sounds like a curiosity, and it's actually the most useful thing to understand about building with AI. A model is one component. The software around it decides much of the result. Some of that software should be thrown away every time the model improves, and some of it should never be thrown away. Most teams can't tell the two apart, and so they keep the wrong half.

What the model actually does

Strip away the product and a language model does one thing. It receives a sequence of tokens and returns a probability for every possible next token. Pick one, append it, ask again. That's how it writes (The Model Never Sees Your Words).

On its own, the model doesn't remember yesterday's conversation, doesn't know today's date, can't open a file, can't send an email and can't tell whether its last answer worked. Every one of those abilities, when you see it in a product, is supplied by ordinary code wrapped around the model. Engineers call that code the harness.

When an AI system does a task with several steps, the harness runs a loop. It assembles what the model will read. The model proposes the next step. The harness parses that proposal, checks whether it's allowed, runs it, records the result and decides whether to go around again.

Scroll sideways to see the whole figure

Look at how little of that loop is the model: only the step where it proposes what comes next. Everything else is written by whoever built the product, and each one is a place where a capable model can be made to look stupid, or a mediocre one helped to look good.

Where the result gets decided

What it sees. The model can only use what's in its context, and the harness chooses what goes there: which instructions, which files, which search results, in what order. Too little and the model guesses. Too much and the one relevant line competes with thousands of irrelevant ones, including stale versions of the truth, like last quarter's price list sitting next to this quarter's. A model asked about a contract clause that was never loaded will answer anyway, and no model, however good, can apply a clause it never saw (What Fits Versus What Gets Used).

What it can do. Here's a technical detail most people never see. A tool, such as "search the database" or "run this code", reaches the model as a paragraph of text in its context describing what the tool does. The model chooses between tools by reading those descriptions, the same way it reads everything else. So forty overlapping tools with vague descriptions are a toolbox where every drawer says "misc", and the model spends its attention deciding which drawer to open.

Vercel found this out with its internal data agent. It cut the agent's tools from 17 specialized ones down to two: a shell and the ability to run SQL against well-documented data. On its test queries, success went from 80 to 100 percent, with fewer tokens and runs about three and a half times faster. The test was small, five queries, so take it as a direction. But the direction is clear: a capable model with a few general tools and good documentation often beats the same model with a drawer full of special ones.

How it loops and checks. Someone has to decide how many steps an agent gets, what happens after an error, and when the work counts as finished. Checking is where many harnesses quietly fail. A model reviewing its own work in the same conversation tends to agree with itself, because the reasoning that produced the mistake is sitting right there, looking reasonable. Research on self-correction finds that models asked to review their own reasoning without new information rarely improve it. A check in a clean context, given only the result and the requirements, works better. A check that needs no judgment at all works best: tests that pass or fail, a total that must add up.

What it may touch. Permissions, credentials, spending limits, who can approve what. In a good harness these live in code the model can't change. In a poor one they live in sentences in the prompt: "never delete customer records", "always ask before sending".

A sentence in a prompt is a request. A permission check in code is a fact.

The difference is invisible until the day the model is confused, or someone hides an instruction inside a document the model reads, which is called prompt injection (Giving an Agent Room to Work).

Settings belong to the harness too. LangChain moved one fixed model from about 53 to about 67 percent on a well-known agent benchmark by changing only its instructions, its tools, the code around the loop and how much reasoning the model was told to spend at each stage. In the same work, the model scored lower at its highest reasoning setting than one notch down. More thinking isn't automatically better, and the dial that controls it sits in the harness.

Models are trained inside a harness

The usual comparison is an engine and a car: same engine, different car, different drive. It's a good start, and it misses something.

When a lab trains a model to act as an agent, it trains it inside some environment: a particular set of tools, a particular format for editing files, a particular loop. Anthropic says it publishes the format of some built-in tools, such as its file editor, and trains Claude on it; that format is built into the model and can't be changed. OpenAI published a file-patch format for GPT-4.1 on which, in its words, the model had been extensively trained, and its GPT-5 models need no description of the patching tool at all because they already know how to write the edits.

So models and harnesses adapt to each other. A harness built around one model's habits can fit the next model badly. That makes every swap a non-neutral test: changing the model while keeping the harness tells you about the pair, and so does the reverse.

It would be tempting to conclude that a model always works best in its maker's tool. The opening result says otherwise: GPT-5.5 did better in Cursor than in OpenAI's Codex, and Anthropic's Opus 4.7 also did slightly better in Cursor than in Claude Code. Nobody has shown which part of the harness made the difference. It could be how each tool picks files, which tools it exposes, its default reasoning setting, or something else entirely.

The practical rule is narrower and more useful than a rule about vendors. Test the combination you will actually run, on work like yours. A benchmark score belongs to a model and a harness together, and a leaderboard tells you what a model did in someone else's harness on someone else's tasks (Build an Evaluation From Your Own Work).

Scaffolding and walls

If the harness matters this much, the obvious move is to build a lot of it. That's where teams go wrong, because harness code comes in two kinds that age in opposite directions.

Scaffolding exists because the model is weak at something today. You split a task into ten rigid steps because the model loses the thread on long ones. You add a function that repairs its broken JSON. You write a paragraph telling it to think step by step. You route each request to one of six specialized sub-agents because one agent with every tool got confused. Each is a reasonable answer to a real weakness.

Walls exist because the world is a certain way. Payments over €10,000 need a second human approver. The agent's database credentials are read-only. Inventory lives in the warehouse system, and this morning's number is the true one. Code can't be merged until the tests pass. Every action is logged with its reason. None of these is about the model.

Now watch what happens when a better model arrives.

Scroll sideways to see the whole figure

The scaffolding that made last year's model usable starts holding this year's model back. The ten-step workflow stops the model reorganizing the work and adds waiting. The "think step by step" paragraph constrains a model that now reasons on its own. The JSON repair function patches a weakness that structured outputs have largely removed. And it's hard to see from inside, because the scaffolding still "works". It just caps the result. The AI researcher Rich Sutton called the long-run version of this the bitter lesson: general methods that use more computation beat methods that encode our cleverness about the problem. Teams rediscover it every few months.

Walls go the other way. A more capable model attempts more, touches more and moves faster, so the approval rule, the read-only credentials and the audit log matter more with each generation.

Anthropic's own engineers made the same point about their harnesses: every component encodes an assumption about what the model can't do on its own, so each one deserves re-testing when the model changes. And harness choices are cost choices. In one of their examples, a full multi-agent harness cost about $200 for a six-hour run against $9 for a single agent working alone. The quality difference was obvious, and so was the bill.

The test for any piece of harness fits in one question:

If the model became perfect tomorrow, would this still be needed?

If not, it's scaffolding. Build it cheaply, write down which weakness it covers, and expect to delete it. When a new model arrives, the first experiment is removing scaffolding.

If yes, it's a wall. Build it properly, in code, where the model can't argue with it. A rule that only exists as a sentence in a prompt is a wish, and wishes are exactly what prompt injection exploits. "Never send an email without approval" is a wall only if the email tool actually checks for approval.

Some pieces need a closer look. A fixed workflow is scaffolding, but a step the process itself requires, such as a legal sign-off, is a wall and stays. Routing to sub-agents is usually scaffolding, unless the split separates permissions, for example a sub-agent that can't see customer data.

Walls are what make freedom affordable

Here's the part most cautious teams get backwards.

Worried about safety, they narrow the model: fewer tools, rigid steps, a human approving every action. That feels careful and usually produces a system nobody uses, because it can't do anything a person couldn't do faster.

When the important limits are walls, you can give the model far more freedom inside them. If a payment can't leave without a second approval, the agent can draft, research and prepare payments freely. If every change is logged and reversible, you can let it change things. Security built into the structure is what lets you stop supervising every step. Security added as caution, step by step, does the opposite.

This is also where "give the model freedom" stops being good advice. A schema validator, a test suite, a rule that totals must reconcile: these are walls too, and they're cheaper, faster and more reliable than asking a model whether something looks right. Freedom belongs to the parts of the work that need judgment. Arithmetic and permissions don't.

Where the value ends up

The second-order consequence matters for anyone deciding what to build or buy.

Models change every few months and can be swapped in days. Scaffolding gets absorbed into the labs' own products, often within a year. What stays slow to replace is what a company owns about itself: its permissions, its sources of truth, its record of how decisions were made and why, its evaluation cases.

So a product that is mostly scaffolding around someone else's model loses ground every time that model improves. A product that owns walls, data access and the trust to use them gains ground, because a better model can do more inside it (Where the Moat Moves). My advice to anyone building on these models: spend your engineering on the second kind.

What to do

When a result disappoints, reconstruct what reached the model before blaming it. Was the needed information in the context? Was it drowned or contradicted by stale versions? Did the model have the tool, and the permission to use it? Did anything check the output before you saw it? Those four questions resolve a surprising share of "the model is dumb" moments. Only then try another model in the same harness, or the same model in another tool.

Choose models by testing the combination. The most useful evaluation set is boring: twenty or thirty real past cases where you already know what good looked like, run end to end in the harness you'll actually use.

Sort your harness. Read the instructions and the code and label every piece scaffolding or wall. Keep scaffolding thin and labeled with the weakness it covers. Before building more, ask whether the lab or a vendor will ship the same thing within a year.

Turn the rules that matter into code. Then give the model room inside them.

Check yourself

Why is a rule written in a prompt weaker than a permission check in code?

A prompt sentence is a request, and a confused model or a hidden instruction can override it. A check in code is a fact the model can't change.

What should happen to scaffolding and to walls when a better model arrives?

Scaffolding should be tested for removal, because it may now cap the result. Walls matter more, because a more capable model attempts and touches more.

Why do walls make more freedom affordable?

If a payment can't leave without a second approval, the agent can draft and prepare payments freely. The structure, not step-by-step supervision, provides the safety.