Leaderboards Measure the Test

On October 11, 2026, Arena's text leaderboard, which ranks AI models by the votes of people comparing two anonymous answers, showed one model in first place and seven more in positions 2 through 8. Their scores ran from 1507 down to 1497.

Next to each rank, the board prints something almost nobody reads: the best and worst rank the model could plausibly hold, given the uncertainty in its score. For all seven of those models, the best case was second place. The board itself was saying that the order inside the group wasn't settled. A week of extra votes proved it. By October 15, Claude Opus 4.6 (high) had slipped from third to fifth, while its score stayed inside its earlier range.

Nobody made a mistake. The ranking was a measurement, and every measurement has a resolution. Read past it and you're reading noise.

That is the whole problem with how most people use leaderboards, so here it is plainly. A leaderboard score measures one test, run one way, graded one way. It is very good at telling a strong model from a weak one and at building a shortlist. It is poor at ordering the models near the top, and it can't tell you which model is best for your work. Use the ranking to decide what to test. Use your own cases to decide what to buy.

A score belongs to five things at once

A lap time belongs to a driver, a car, a set of tires, a track and a timing system. Quote it without those and nobody can tell whether it predicts anything about your road. A benchmark score, a model's result on a fixed set of test tasks, has exactly the same structure, except that the conditions behind it usually have to be dug out of footnotes.

Take one real row. On October 11, the Terminal-Bench 4.0 board, a benchmark for AI agents working in a computer's command line, put Claude Opus 5.5 at the top: 64.8 percent of tasks solved, plus or minus 3.1, over 330 trials, at about $14.30 a trial. That single number belongs to five things.

Scroll sideways to see the whole figure

The model is a specific release, and often a specific configuration of it. The harness is the software around the model: its tools, its instructions, its loop and its rule for stopping. Changing only the harness has moved one model by more than 25 points on the same tasks (Same Model, Different Results). The settings include, above all, the effort setting, which tells a reasoning model how much to think before it answers. The task set is the questions: how many, how old, and whether they have been public long enough to leak into training data. The grader turns an attempt into pass or fail, and it can be a program, another model or a crowd of voters.

The settings alone are bigger than most people assume. On that same board, Claude Opus 5 appears five times, once per effort setting, and its score runs from 34.9 percent at the lowest to 53.9 percent at the second highest. That 19-point spread inside one model is larger than the 13 points between Opus 5 and Opus 5.5. More thinking doesn't always pay, either: Opus 5 at its maximum setting scored slightly below the notch beneath it, at similar cost (What Thinking Buys). So when a vendor's launch table runs its own model at one setting and a rival at another, it is comparing dials as well as models.

Change any slot and you get a different number. Your own work changes all five at once: your tasks, the harness you build or buy, your settings, and a grader who knows what good looks like. A leaderboard transfers to you only as far as those slots match.

How a test stops measuring

Economists have a name for what happens next. Goodhart's law, in its familiar wording, says that when a measure becomes a target, it ceases to be a good measure. A benchmark is published so the field can compare models. Once the comparison matters, everyone optimizes for it.

The cleanest measurement of how much came from Scale AI in 2024. Its annotators wrote 1,205 new grade-school math problems in the style of GSM8k, a famous public benchmark, so close in style that other people couldn't pick out the new ones better than chance. Then dozens of models took both tests under identical conditions, so any difference had to come from the questions.

Some models had clearly studied for the old test. A small open model, Yi-6B-Chat, fell 8 points, and several other open models lost 4 to 7. The frontier models of the time barely moved: GPT-4o went from 93.1 to 92.9 percent.

The interesting part is how a model studies for a test. The obvious route is leakage: test questions end up in the training data, a problem called contamination. But the authors found that leakage explained only part of the pattern, and suggested a quieter route that needs no wrongdoing at all. Suppose a team trains ten candidate versions of a model and ships the one that scores best on a famous benchmark. It is selecting for real ability, and also for luck and for fit to that particular test. The winner will look better on that benchmark than it is anywhere else, and the effect grows with the number of candidates and the noisiness of the test.

Studying for the test needs no leak. Choosing the version that scores best on it is enough.

Tests also wear out from the top. Once the leading models crowd near the ceiling, a condition called saturation, the test's own flaws start deciding the order. MMLU, a broad knowledge test from 2020, had models above 90 percent by early 2025, and a later re-check estimated that roughly one question in fifteen has a wrong or ambiguous answer key. Two models at 91 and 93 percent can then be separated mostly by which flawed questions each happened to answer the way the key did. The difference is repeatable, and it says little about ability.

That is why benchmarks are replaced so quickly, and why scores don't carry across versions. A February 2026 result discussed in Same Model, Different Results used Terminal-Bench 2.0. By September, the live board was on version 4.0. The remedy is a private test set, which nobody can study for. It still ages.

Votes measure preference

Arena avoids answer keys altogether. A person writes a prompt, sees two anonymous answers and picks the better one. A method borrowed from sports ranking, the Bradley-Terry model, turns millions of those votes into a strength score for each model.

Two things follow, and both matter more than the rank column.

The first is how small the gaps are. On Arena's scale, the 18 points separating first and second place on October 11 meant the leader would be preferred in about 52.6 percent of head-to-head votes. Telling a 52.6 percent coin from a fair one takes a lot of flips, and the leader, flagged as a pre-release model, had fewer than 5,000 votes, against about 79,000 for the model in third. Its range of plausible scores was about two and a half times wider.

The second is what gets measured, which is preference. People tend to prefer longer, more formatted, more confident answers, and that is a different property from being right. When Arena's team estimated the effect of length and formatting on votes in 2024, length mattered most by far, and correcting for it moved models up and down the board. That correction, style control, is now the default view. It can only adjust for the features someone thought to measure.

Then there's selection. A 2025 audit of the board, published as "The Leaderboard Illusion", found that some providers could test many private variants and publish only the best, as many as 27 from Meta in a single month. Even if all 27 had been identical, the best of them would look better than it really was, purely from noise: for scores as precise as those at the top of the board, about 3 to 9 points better. That is comparable to the entire 11-point spread across positions 2 to 8. Arena has changed its procedures since, so the audit describes early 2025. The arithmetic of picking the best of many noisy scores hasn't changed.

Preference is a perfectly good target for some work, such as a chat assistant answering open questions. It is beside the point for others, such as pulling fields out of invoices.

The error bars everyone skips

Underneath all of this sits one piece of simple statistics, and it is the part leaderboard readers skip.

About 73 percent. Roughly 3 to 1 odds: some evidence, and well short of a ranking.

A score is an estimate. The test's 500 items are a sample from the much larger population of tasks the test stands for, and a different 500 would have produced slightly different scores. The range of true pass rates that could plausibly have produced each result, called a 95 percent Wilson interval, is about seven points wide for both models, and the two ranges overlap by more than five points. The 1.6-point gap sits comfortably inside the noise.

Scroll sideways to see the whole figure

There's a subtlety that matters even more. Both models answered the same 500 items, so most of those items carry no information about which model is better: the ones both pass and the ones both fail cancel out. Only the items where the models disagree carry evidence. If they disagree on just 12 items and A wins 10, A is very probably better. If they disagree on 100 and A wins 54, you've learned almost nothing. The headline scores are identical in both cases, and leaderboards almost never publish the disagreement count.

Overlapping intervals leave a ranking unresolved, which is a weaker finding than a tie.

How big would a test have to be to settle a small gap? Reliably separating a true difference of 1.5 points takes about 11,700 items per model, far beyond the size of most benchmarks. So most of the gaps inside the top group of any ranking are smaller than the test can detect. Real boards show it. On Terminal-Bench 4.0, six entries in ranks 3 through 8 sit at either 58.18 or 57.88 percent. Three tenths of a point apart, with error bars of about three points each, they are ranked as if the order meant something.

What rankings are good for

None of this makes benchmarks useless. A noisy instrument with a known resolution is useful for what it can resolve, and three things clear the bar.

Large differences. On Terminal-Bench 4.0, Fable 5.1 scored 57.9 percent and GLM-5.3, from the middle of the board, 41.8. A 16-point gap is larger than every source of noise above.

Direction on a fixed test. Opus 5 at maximum effort, released in July, scored 51.8 percent. Opus 5.5 at maximum, released in September, scored 64.8. Thirteen points on the same version of the same test is a real improvement.

Cost. The column most readers ignore is often the most decisive one. On that board, Opus 5.5 costs about $14.30 per trial and GPT-6.1 Sol about $1.92: more than seven times as much for 6.7 more points. Whether those points are worth it depends on what a failed task costs you, and no board knows that.

Agreement across boards is worth something too. When rankings built on different tests, graders and harnesses agree on who's strong, their independent errors tend to cancel. It still falls short of proof for your use, because labs aim at the same visible benchmarks and several boards share the same grader models.

How I'd read one

My advice is to read a leaderboard bottom up, the way you'd read a lab report, and it takes about two minutes.

Date and version first. A benchmark name without a version, or a table without a date, is the first warning. A score near 90 percent on an old public test belongs to a crowd.

Then the tasks. Look at five sample items. If they don't resemble your work, the score measures general capability at best, which is fine for screening and useless for choosing.

Then who ran it, and how. Some organizations run their own evaluations of everyone's models. A vendor's launch table reports runs at settings the vendor picked. Find the effort setting and the harness in the footnotes.

Then the error bars, before the ranks. Treat two scores as separated only when the gap is larger than about one and a half times the half-width of the wider interval. Treat a leader with few votes or trials as provisional: a top score picked from a noisy field tends to drift back toward the pack as data arrive.

Then the cost column.

What most people get wrong comes after all that: the jump from "best on this test" to "best for us." It is the conclusion readers draw most often and the one the data support least. When options are tens of points apart on a board that looks like your work, a ranking is a reasonable basis for a first pick. When they're within a few points, or a mistake costs real money, the ranking has done its job once it hands you three or four candidates.

Then test those candidates on about thirty of your own real cases, with known good outcomes, run through the harness you'll actually use and graded the way you grade (Build an Evaluation From Your Own Work). It is the only test that fills all five slots with your contents.

Use the ranking to decide what to test. Use your own cases to decide what to buy.

There's a second-order effect worth naming for anyone managing a team. When your people tell you the board's leader is worse for their work, check the five slots before you doubt them. They may be the only ones in the building measuring the right thing.

Benchmarks will keep being replaced, and next year's leader will sit at the top of a test that doesn't exist yet. The question a leaderboard answers, which model did best on this test, will stay narrower than the question you're asking. Closing that gap is your job, and it is cheaper than it sounds.

Check yourself

Why can one model's score swing by 19 points on a single board?

The effort setting is part of the score. Opus 5 appears once per setting, and the spread between its lowest and second-highest setting is larger than the 13 points between Opus 5 and Opus 5.5.

Why can a model look better on a famous benchmark without any leaked questions?

A team that trains many candidate versions and ships the best scorer selects for luck and fit to that test as well as real ability. The winner looks better there than it is elsewhere.

Why does a 1.6-point gap on a 500-item test not settle which model is better?

A score is an estimate from a sample, and both ranges are about seven points wide and overlap. Only the items where the models disagree carry evidence.