Build an Evaluation From Your Own Work

On September 30, 2026, Anthropic told developers that a snapshot of one of its models, Sonnet 4.5, would be retired on November 30. Sixty-one days' notice, a day more than the minimum it commits to.

Picture two companies running a customer process on that model. The first spends the two months arguing. Someone tried the replacement and thinks it's worse at the tricky emails. Someone else thinks it's fine. A vendor says its own tool would solve the problem. Nobody can settle it, because nobody has anything to settle it with.

The second company opens a file of thirty real cases from its own work, each with a known good answer, runs them through the new model in the afternoon, reads the failures, and decides by Friday.

The difference between those two companies is the whole argument of this piece. The only reliable answer to "is this model, prompt or tool better for us?" is a small evaluation set: your own real cases with known good outcomes, run through the whole system after every change. Public rankings help you build a shortlist (Leaderboards Measure the Test). Vendor demos show you what a product does on inputs its team chose. Neither samples your customers, your suppliers, your phrasing or your exceptions. Your file does.

Building one is less work than most teams assume, and reading it honestly is harder. Both parts are below.

Why an evaluation is not a test suite

Engineers arrive with a good instinct: pin behavior down with tests that run on every change. They also arrive with an assumption that a language model breaks. A traditional software test is deterministic: one input, one correct output, the same result every time.

A model gives you none of that. The same input can produce different outputs on different runs. Many different outputs can be acceptable. And deciding whether an output passed often needs judgment. So the result of an evaluation is a rate with uncertainty around it, and the practice looks more like quality control in a factory than like unit testing. You sample, you count defects against a specification written in advance, and you decide whether to accept the lot.

Scroll sideways to see the whole figure

The file

Take a concrete job, constructed for illustration. An accounts-payable team receives emails from suppliers saying an invoice is wrong or unpaid. Software around the model (the harness, which Same Model, Different Results explains) fetches the matching purchase order and receiving record. The model then classifies the dispute, extracts the invoice number and the amount claimed, computes the gap against the purchase order, and drafts a reply that a clerk reviews and sends. The team wants to know whether a cheaper configuration can replace the current one.

Start from closed disputes, because their outcomes are known: the right category, the right amount, a reply the clerk approved. Check those answers first. Clerks make mistakes, and a wrong "known good" answer punishes the better system.

The mix matters more than the count. A sensible thirty might be 15 normal cases, sampled at random from closed disputes; 9 hard ones, picked on purpose from disputes that took a clerk several rounds (two invoices in one email, a partly applied credit note, a supplier writing in another language); and 6 adversarial ones, such as an email containing an instruction aimed at the model, or a request to change bank details.

Why not sample everything at random? Because random samples miss exactly the cases that do damage. If 3 percent of disputes involve a tricky multi-invoice email, a random thirty contains none of them about 40 percent of the time. The price of choosing edge cases deliberately is that your pass rate is no longer your production pass rate. That's fine, as long as you report the normal, hard and adversarial groups separately instead of blending them into one number.

Have a second person label ten of the cases independently. Where two competent people disagree on the right answer, the case is ambiguous, and it waits until the policy is clarified. And keep the file yourself. OpenAI's hosted Evals product goes read-only on October 31, 2026 and shuts down a month later, which is a useful reminder that a vendor's platform is a poor place for the only copy of your cases.

Decide what "good" means before you look

Write the checks before you see any outputs. People, and models, drift toward approving whatever is in front of them.

Each check should be answerable yes or no. Is the category right? Is the amount exact, to the cent? Does the reply cite the invoice number and name the next step? Does it avoid promising a payment the policy forbids? Does it ignore the instruction hidden in the email?

Then split the checks into two kinds. Gates are failures that cost money or trust: a wrong amount, an unauthorized promise, obeying an injected instruction. Quality covers tone and completeness. Never average the two. An average lets a pleasant reply hide a wrong number, and a wrong number is the thing that matters.

Who grades

Use the cheapest grader that can actually see the property you care about, and spend human attention on what's left.

Code first. A check that the gap reconciles to the cent catches a whole class of error without anyone reading anything. Code checks are cheap, exact and repeatable, and blind to everything they weren't written for.

People second. Someone who knows the work is the best judge of stance and tone. At thirty cases the cost is modest, roughly an hour of reading at a couple of minutes per draft. It stops being modest when you run thirty cases three times on two configurations: 180 drafts, about six hours, for one comparison.

A model as judge, third, and only after calibration. Another model can grade at scale, and it has documented habits. In the original study of models as judges, from 2023, judges shown the same two answers in swapped order often changed their verdict, mostly favoring whichever came first; GPT-4 gave the same verdict both ways in only 65 percent of cases. Judges also tend to prefer longer answers, and some favor their own writing over equally good answers from others.

So treat a model judge like any instrument: calibrate it. Have a person grade 40 outputs, run the judge on the same 40, and compare. Look past the agreement rate. Suppose the person passes 31 and fails 9, and the judge agrees on 35 of 40, or 87.5 percent. That sounds good until you notice that a judge stamping "pass" on everything would agree 77.5 percent of the time. The number that matters is how many of the 9 real failures the judge caught. Here it caught 5. That makes it a decent filter and a poor gatekeeper. And give the judge the reference answer: in that same 2023 study, a judge given the correct answer was wrong on 3 of 20 math gradings, against 14 of 20 without it.

Run the whole system, more than once

Don't test the model in a playground. Test what you'll deploy: the retrieval that fetches the purchase order, the instructions, the tools, the output parser, the retry rules. A failure can come from the harness (the lookup fetched the wrong purchase order), and a model-only test would never see it.

Change one thing at a time. If the new configuration differs in both model and prompt, a change in results can't be assigned to either. Record cost and latency beside quality, because passing two more cases at four times the price is a different decision from passing two more at the same price.

Then run each case three to five times, because one run is one draw. How you summarize depends on how the output is used. If a clerk reads every draft, the question is whether a good draft usually appears. If the output goes out unreviewed, the question is whether every run succeeds, and that falls fast: a case that passes 90 percent of the time passes three runs in a row only 73 percent of the time. Pick the summary before the results arrive.

Reading the result without fooling yourself

Not very.

With thirty cases, each pass rate comes with a wide range of plausible true values, which statisticians call a confidence interval. Ninety percent on 27 of 30 is compatible with a true rate from about 74 to 97 percent. Eighty percent on 24 of 30 is compatible with about 63 to 91. The ranges overlap heavily.

Scroll sideways to see the whole figure

The better check uses the fact that both configurations ran the same cases. Line them up case by case. In this illustration, 22 cases pass under both and one fails under both, so those 23 tell you nothing about which configuration is better. Only the seven cases where they disagree carry information: the current one won five and the cheaper one won two. If the two were equally good, a split at least that lopsided would happen about 45 percent of the time. Six wins to zero would be convincing. Five to two is not.

Only the cases where the two configurations disagree carry information about which is better.

That sets honest expectations. Thirty cases will catch large gaps and regressions. They won't rank near-equal options: telling a true 90 percent from a true 80 percent reliably takes about 200 cases per configuration. When the numbers don't separate the options, decide on other grounds (price, latency, the kinds of failures each one makes), and write that decision rule down before you look, so nobody can reinterpret it afterward to suit a preference.

Read the failures

The score says how many failed. The failing outputs say why, and the why is what you can act on. Print every failure beside its case and the reference answer, write one line on what went wrong, group the lines, and count.

A failure comes from one of five places, and each needs a different fix: the model (it misread a credit note), the harness (it fetched the wrong purchase order), the grader (a check rejected a correct reply worded differently), the label (the clerk's recorded answer was wrong), or the specification (two sensible people disagree about the right answer). Some of the fixes that move the score most touch no model at all. Correct a wrong label and both configurations improve; fix the purchase-order lookup and the one that tripped on it does.

There's a trap here. Once you edit the prompt to fix the failing cases, those cases stop testing it. Lock away about a third of the set: look at their scores, never use their transcripts for fixing, and refresh them with new real cases each quarter.

Keep it alive

An evaluation built for one decision goes stale the week after. What keeps it useful is a few dull habits.

The ratchet. Every failure that reaches production, whether a clerk's correction, a supplier complaint or an escalation, becomes a new case with the correct outcome before anyone fixes it. The set grows toward the failures your system actually has. This is how work starts to compound instead of repeating itself (When Work Starts to Compound).

Triggers. Rerun the set on every change to the prompt, the model, a tool, the harness or the data it reads, and monthly on an unchanged setup. Providers repoint model names and vendors change hosted tools; your configuration can change without you touching it, and only a scheduled run notices.

An owner. One named person owns the cases, the grader versions and the results log, and every result carries its date and configuration, so March can be compared with October.

Guarding the number. A pass rate that becomes a target gets optimized until it stops describing the work. Keep two or three numbers, keep adding cases, and every so often have someone read ten raw failures aloud. A team that only watches the number slowly loses sight of what it stands for.

Be honest about what the file can't tell you. It covers the cases you thought of. A failure that happens 1 percent of the time won't show up in thirty cases about three times out of four, so rare, severe risks need other controls such as permissions and review (Giving an Agent Room to Work). And a higher pass rate is not the same as faster dispute closure or lower cost per resolved dispute, which have to be measured on the business itself.

An evaluation measures what you thought to check. Everything else remains unmeasured, and the report should say so.

Start with ten

The usual objection is time, and for most teams it's real. My answer is a smaller version, built now.

Ten cases fit in an afternoon: five normal, three hard, two adversarial, pulled from last month's closed work. Code checks cover the two or three things that have exact answers. A person reads every output. One file holds the cases, and the file runs before every change. Ten out of ten proves less than it sounds (the true rate could still be as low as about 72 percent), so the small version rules out big problems and says nothing about reliability. It's also the seed: the thirty-case set grows from it, one real failure at a time.

The other objection is that an experienced person's judgment is cheaper and usually right. I agree with the premise. An experienced clerk reading outputs is an evaluation. The file is that judgment written down so someone else can rerun it on a new model without the expert in the room, compare it with last quarter, and keep it when the expert leaves.

That's the second-order reason to build it. Models change every few months, and every change is a decision. A company with a file makes those decisions in an afternoon, with evidence about its own work. A company without one makes them by argument, and keeps making them that way.

Check yourself

Why is an evaluation more like quality control than a unit test?

The same input can give different outputs, many outputs can be acceptable, and judging often needs judgment. So the result is a rate with uncertainty.

In the 27 versus 24 example, why isn't the lead convincing?

Only the cases where the two configurations disagree carry information. A five to two split would happen about 45 percent of the time if they were equally good.

Why lock away about a third of the set?

Once you edit the prompt to fix the failing cases, those cases stop testing it. Held-back cases keep the measurement honest.

from math import sqrt, comb

Z = 1.959963984540054

def wilson(k, n, z=Z):
    p = k / n
    d = 1 + z * z / n
    center = (p + z * z / (2 * n)) / d
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return center - half, center + half

def sign_test_two_sided(a_only, b_only):
    m, k = a_only + b_only, min(a_only, b_only)
    return min(1.0, 2 * sum(comb(m, i) for i in range(k + 1)) / 2 ** m)

print(wilson(27, 30))              # (0.7438, 0.9654)
print(wilson(24, 30))              # (0.6269, 0.9049)
print(sign_test_two_sided(5, 2))   # 0.4531