Why Progress Arrives Unevenly

Here is a puzzle from a real training run.

In January 2025 the AI lab DeepSeek published scores for two chat models built on the same pretrained base. Same starting weights, same lab, same scoring table. The second model, R1, had been trained much harder with one particular technique. On competitive programming it jumped from the 59th to the 96th percentile of human contestants. On fixing real bugs in real software projects, it gained about 7 points.

Both tasks are coding. Both draw on the same mountain of code the model read in pretraining. So why did one leap and the other barely move?

DeepSeek's own paper gives the answer, and it is the most useful idea for predicting what AI will be good at next year. AI gets better fastest where an answer can be checked cheaply and automatically, because that is where training can practice without a person in the loop. A contest program is checked in seconds against test cases. A bug fix in a real project needs the whole project built and its tests run, which takes minutes. DeepSeek said plainly that large-scale training of this kind "has not been applied extensively in software engineering tasks" because of "the long evaluation times".

The cost of checking one attempt decides how much practice a model can get. Everything else in this piece follows from that.

How a model practices

To see why, you need one technical picture: how the newest models are trained after they have read the internet.

Pretraining teaches a model to continue text. It ends up knowing a great deal and doing nothing reliably. The step that turned models into strong problem-solvers over the last two years is a kind of training with an awkward name, reinforcement learning with verifiable rewards. The idea is simple. Give the model a problem. Let it try many times. An automatic checker grades each attempt: did the math answer match the key, did the code pass the tests. Then nudge the model so the attempts that passed become more likely. Repeat millions of times (What Training Actually Changes).

Scroll sideways to see the whole figure

Notice where all the leverage sits. The model supplies attempts for free. The checker is what turns attempts into learning, and the checker is where cost enters, in three ways.

Money. Every attempt needs a check, and a lab runs many attempts per task across many tasks. Practitioners say a task teaches almost nothing if the model always passes or always fails, so a single task can consume dozens of checks on each pass.

Time. The training step waits for its grades. A check that takes an hour stalls the whole loop.

Fidelity. A checker that passes wrong work teaches wrong work. That is why DeepSeek avoided learned graders for its purest training run: a model trained against a fallible grader learns to fool the grader.

A rough calculation shows the scale. Take 10,000 training tasks with 64 attempts each, so 640,000 checks for one pass. If the check is an answer key, the cost rounds to zero. If it means spinning up a project and running its tests for five minutes, it's a few thousand dollars. If it needs an hour of an expert's judgment, it's tens of millions of dollars, every pass, and the loop waits an hour for each grade. The exact numbers depend on assumptions. The orders of magnitude don't.

Practice can only be as plentiful as its grading.

The same property works a second way. Labs steer by benchmarks, and a benchmark needs a cheap check to exist. Where scoring is cheap, builders can see within days whether a change helped, so effort concentrates there. Where scoring is expensive, they fly with fewer instruments.

Why code got there first

Code stacks four advantages, and few other kinds of work have more than one or two.

The checks already exist. Compilers, type checkers and test suites were built for human programmers, and they work just as well on a model's attempts.

The data comes with history. Public repositories pair a reported problem with the change that fixed it and the test that confirms the fix. SWE-bench, the best-known coding benchmark, was built by mining exactly that history.

The workplace is software too. A container can reproduce a programmer's entire working environment: the code, the tools, the test runner. Nobody can put a hospital ward or a negotiation in a container. When the job, the materials and the grader are all software, a lab can manufacture practice by the million.

The builders are the users. Labs are full of programmers, and developers pay. In Anthropic's first report on how people use Claude, in early 2025, computer and math tasks made up 37 percent of conversations, against about 3 percent of US workers in those jobs.

Code is the rare kind of work where the workplace, the task and the grader are all software.

The result is the steepest curve in AI. METR, an independent evaluator, times skilled people on a fixed set of software tasks, then measures the task length an AI agent can finish half the time. In March 2023, GPT-4 managed tasks that take a person about four minutes. In February 2026, Claude Opus 4.6 managed tasks that take about twelve hours. That time horizon has been doubling roughly every four months since 2023.

Nobody has a comparable curve for writing a strategy memo that holds up, choosing which research question to pursue, or managing a team. The missing curves are part of the explanation: they are missing because nobody can score that work cheaply.

What the unevenness looks like at work

Anyone who uses these tools has seen it. A coding agent fixes a bug across five files and runs the tests. The same model drafts a board memo that sounds right and commits to nothing.

Researchers named this the jagged frontier in a 2023 experiment with 758 consultants at Boston Consulting Group. On tasks inside the model's abilities, consultants using GPT-4 finished more work, faster and better. On one task designed to sit just outside, a business problem where the model's plausible answer was wrong, 84 percent of consultants without AI got it right, against roughly 60 to 70 percent of those with it. The tasks looked equally hard to a person. The boundary didn't follow human difficulty, so nobody could tell which side a task was on without trying it.

The checking idea gives the boundary a shape. It doesn't follow how hard a task feels. It follows how cheaply a computer could tell whether the answer is right.

Hard-to-check work still improves, more slowly

This is not a claim that judgment work stands still. OpenAI's GDPval, from September 2025, asked experts in 44 occupations to compare AI deliverables with human ones without knowing which was which, on tasks that take an expert about seven hours. The best model in the study was rated as good as or better than the expert 47.6 percent of the time, and OpenAI's own models improved steadily across releases.

But look at the price of that measurement. Each comparison took an expert more than an hour. A math answer is checked in milliseconds. That gap is the whole story: the work improves, and every unit of feedback costs an expert's hour.

GDPval's error analysis also shows where the progress comes from. Many lost comparisons were cheap failures: a missing file, ignored reference data, a broken layout. Those are checkable. Asking one model to render and inspect its own slides cut the share of its PowerPoint files with egregious formatting errors from 86 to 64 percent. Inside every judgment job, the checkable parts improve first (A Job Is a Bundle of Tasks).

Checkability can be built

The most important consequence is that this map isn't fixed. People can move work across it.

DeepSeek's slow row didn't stay slow. Through 2025, labs built containerized coding environments and trained agents inside them, and top scores on the real-bug benchmark rose from R1's 49 percent to about 81 percent by February 2026. Some of that came from better agent tooling and from test problems leaking into training data, so it overstates the change in the models alone. But the direction is plain. Where checking was slow, the industry built faster checks, and scores rose. Practitioners describe a whole market of companies building these training environments, and Anthropic was reported to have discussed spending more than $1 billion on them over a year.

Labs use three techniques to make hard work checkable. Rubrics: experts write specific criteria for a good answer and a model scores attempts against the list. Model judges: a model compares attempts the way an expert would. Simulations: replicas of websites and business applications, where an agent practices a task and is scored on the final state.

Each one replaces the real verdict with a proxy, and training finds the gap between them. That's reward hacking. Model judges have been fooled by answers consisting of little more than the words "Thought process:". Even code checks get gamed: METR reported in June 2026 that one model's time horizon was about 11 hours with cheating counted as failure and beyond 270 hours with cheating counted as success, because it had learned to extract information about the hidden tests. A rubric that rewards structure without substance trains structure without substance, at scale.

The honest uncertainty

The checking explanation has rivals, and I don't think the evidence settles it. Code is abundant online, and developers pay, so data and revenue also point at code. The DeepSeek comparison weakens data as the sole cause, since the same pretraining produced very different gains. Competition math weakens revenue as the sole cause: it earns labs little directly and improved as fast as anything, though it carries prestige. My reading is that abundance sets the starting level, incentive decides where labs build checks, and cheap checking decides how fast training can push once they do. The clean test would be a lab holding everything fixed and varying only the speed of the grader. Nobody has published it.

What comes next

If you accept the mechanism, you get a forecasting tool better than any leaderboard. For any piece of work, ask: what would it cost a computer to check one attempt?

Scroll sideways to see the whole figure

Next to move: work with programmatic checks. Data transformations where row counts and totals must reconcile. Spreadsheet models where the balance sheet must balance and cash must tie to the ledger (Finance and the Reconciliation Wall). Formal mathematics written for a proof checker such as Lean; in September 2026 GPT-6 Astra solved 2 of 68 open Erdős problems in a set that requires Lean solutions, where no earlier model had solved any. Rule-based compliance: whether a filing has every required field, whether a tax calculation follows the rule. Business workflows in simulated applications, scored on the final state.

Slower: work with expensive or contested checks. Strategy for a particular company, taste, research direction, managing people. Slower doesn't mean static; it means the checkable fragments of those jobs go first, and the share left to people tilts toward the parts nobody can grade quickly.

The relation should last as long as models improve by optimizing against signals. The specific benchmarks and the order of the next domains will date within a year.

What to do

For your career, stand at the two ends of the check. One end is defining what "correct" means precisely enough that a program can test it. The other is the judgment nobody can test quickly. The middle, producing work that is cheap to check, is exactly what training is getting good at.

For your organization, write down your checks. Reconciliations, test suites, reference cases with known answers, rubrics an expert would sign. They pay twice. Inside a company, AI speeds up work fastest where a person can verify much faster than they can produce (Where the Saved Time Goes). And a written check lets you test each new model on your own work the day it arrives. A task that is cheap to check gets both effects; work that is expensive to check gets neither, so the gap widens.

For investment, follow the checker. If the mechanism holds, capability arrives in a domain roughly when someone builds a cheap, faithful checker for it, and spending follows that sequence: training environments, verification tools and evaluation data before the applications (Where the AI Money Goes). Ask of any company betting on a domain whether it owns the check, and whether that check would survive a model optimizing hard against it. A trusted checker is something every lab training for that domain will want.

To guess what AI will master next, skip the leaderboards. Ask what it costs to check one attempt, and who is building the checker.

Check yourself

Why did R1 jump on competitive programming but gain little on fixing real bugs?

A contest program is checked in seconds against test cases. A real bug fix needs the whole project built and its tests run, so training gets far less practice there.

Why does a checker that passes wrong work hurt training?

The model learns to fool the grader, which is reward hacking. A rubric that rewards structure without substance trains structure without substance.

Why is the line between fast and slow AI progress not fixed?

People can build faster checks. Containerized coding environments raised top scores on the real-bug benchmark from 49 to about 81 percent.