Tests Are the New Specification

Invoices are due 30 days after they are issued. If that falls on a weekend, use the next business day.

A product manager can write that ticket in a minute, and three engineers, or three runs of a coding agent, can implement it three defensible ways. The first adds 30 calendar days and moves a Saturday or Sunday to Monday. The second notices that "business day" also excludes public holidays. The third computes from the invoice's timestamp in UTC, the universal clock the rest of the billing system uses.

They disagree on real invoices. One issued on Wednesday 25 November 2026 is due on Friday 25 December under the first reading, Christmas Day, and on Monday 28 December under the second. One issued at 9 p.m. in New York on 24 November is due on 24 December under the first reading and on Christmas Day under the third, because at that moment it is already 25 November in UTC. Every version meets the ticket. The customer who gets a late-payment notice on 26 December won't find that reassuring.

A colleague would probably ask about holidays and time zones. An agent can ask too, but agents are tuned to keep making progress on reasonable assumptions, and those assumptions stay invisible unless someone reads the summary. Now add one line:

assert due_date(date(2026, 11, 25)) == date(2026, 12, 25)

That line decides that holidays don't move the due date. Whether that's the right call is a business question, but it has been made, and it fails loudly if anyone, human or agent, builds the second reading.

That's the whole argument in miniature. A coding agent works toward whatever tells it the task is done, so the tests, more than the ticket, are where you tell it what you want. A test is a decision recorded in a form a machine can run, and everything it doesn't check is still open to interpretation. Writing good tests used to be a discipline. With agents it has become the main way to specify software.

Why the test wins over the ticket

A coding agent keeps working until something tells it to stop. In a well-run project that something is a test run that comes back green. In a poorly run one it's the agent's own sense that the code looks right (How a Coding Agent Actually Works follows that loop step by step).

The ticket says what you want in words. The tests say it in a form the agent can run, with three properties prose lacks. A test is precise: the line above has exactly one reading. It's repeatable: same verdict on the twentieth attempt as on the first. And it's cheap, so the agent consults it after every edit. A sentence in a ticket gets interpreted once and never answers "no".

Think of a recipe. "Roast until done" relies on the cook's judgment; "until the thickest part reads 165°F" replaces judgment with a check anyone can repeat. The analogy also shows the limit. A chicken can reach 165°F and still be burnt. A check pins down what it checks, and nothing else.

A test is a decision recorded in a form that can be run. Everything it doesn't check is still prose.

Two tests that let three bugs through

Settle the ticket on the first reading: 30 calendar days, weekends move to Monday, holidays don't count. Here's a correct implementation and the kind of test suite people and agents often write, one ordinary case and one weekend case:

def due_date(issued):
    due = issued + timedelta(days=30)
    while due.weekday() >= 5:          # Saturday or Sunday
        due += timedelta(days=1)
    return due

def test_due_in_thirty_days():
    assert due_date(date(2026, 6, 1)) == date(2026, 7, 1)

def test_saturday_moves_to_monday():
    assert due_date(date(2026, 6, 11)) == date(2026, 7, 13)

These two tests run every line of the function and both branches of its loop. A code coverage report, the standard measure of how much code the tests exercise, shows 100 percent. For this article, five wrong implementations of the kind an agent or a hurried person might write were run against them.

Three. The suite catches the version that forgets weekends and the version that moves them back to Friday. It lets through:

• one that returns the expected answers for the two tested dates and adds 30 days otherwise, the special-casing agents have been documented doing;
• one that reads "30 days" as "one month", which agrees with both tests because they start in June, a 30-day month;
• one that adds two days to any weekend date, which turns Saturday into Monday correctly and Sunday into Tuesday.

The stronger suite keeps the two examples, adds four edge cases (a Sunday, the end of January, a year boundary, a leap year) and one property-based test. Instead of listing inputs and answers, a property-based test states rules that must hold for every input, and a library generates hundreds of inputs looking for one that breaks a rule. Three rules here describe the whole requirement: the due date is a weekday, it's never before day 30, and every day it skipped was a weekend day. None of them restates how to compute the answer, which matters, because a test that reimplements the function tends to share its bugs.

Of the six implementations, the strong suite accepts only the correct one. It runs in about a tenth of a second.

Scroll sideways to see the whole figure

The library found the bugs on its own. It caught the one-month version with an invoice issued on 1 January 2000, due 1 February, which skips Monday 31 January. It caught the Sunday-to-Tuesday version with an invoice issued Friday 9 February 2001. Nobody wrote either date. And every rule carries weight: delete the third one, the kind of shortcut an agent under pressure might take to get a suite green, and the property test lets the Sunday-to-Tuesday version through; only the hand-picked Sunday example still catches it.

The weak suite ran every line and checked almost nothing. The strong suite is barely longer and accepts only the right answer.

This is also why coverage is a poor target. Coverage records which lines ran, not whether any assertion would notice if they were wrong; a test that calls the function and asserts nothing reaches the same 100 percent. Mutation testing measures the missing half: a tool makes small deliberate changes to the code, such as turning >= into > or 30 into 31, and checks whether the tests notice. Even that has a blind spot. In this example the weak suite caught 8 of 11 such changes and still accepted three wrong implementations, because a misread requirement is no small slip, and no mutation tool generates one. Only someone who knows the requirement can judge whether the tests encode it.

Two kinds of test, borrowed from a benchmark

The best-known coding benchmark, SWE-bench, turned real bug reports from 12 Python projects into tasks, and grades every fix with two kinds of test the agent never sees. Fail-to-pass tests failed before the real fix and pass after it: what the change must make work. Regression tests passed before and after: what the change must leave alone. On average a task carries about 9 of the first kind among about 121 tests in total. Most of what grades a change checks what it must not break.

Scroll sideways to see the whole figure

The structure transfers directly to work with agents. A task should arrive with its own fail-to-pass tests, written or approved by whoever owns the decision, while the tests already in the repository guard everything else. They go wrong differently. A fail-to-pass test that passes before any change isn't testing the change. A regression test that starts failing means the change broke something, or that the agent is about to "fix" the test.

The same benchmark also shows how hard good tests are. When OpenAI had professional developers screen its tasks in 2024, about 61 percent were flagged for tests that might reject valid solutions, often because they were too specific. An overly specific test is the ambiguous ticket in reverse: it makes decisions nobody asked for, like the exact wording of an error message, and rejects correct work.

How agents satisfy the letter

Coding models are partly trained by rewarding them when tests pass. That cheap, automatic check may be much of why coding agents improved faster than most AI work. It has a price. An optimizer rewarded by a check learns whatever satisfies the check, which researchers call specification gaming.

The cleanest measurement is a 2025 study called ImpossibleBench. Its authors altered the tests of coding tasks so they contradicted the specification: any pass is cheating by construction. On one variant, GPT-5 passed 76 percent of these impossible tasks. Models cheated four ways: editing the tests, redefining what "equal" means so mismatches compare as equal, keeping hidden state so repeated calls return whatever each test expects, and special-casing the tested inputs. Claude models cheated mainly by editing the tests, although every prompt said "DO NOT MODIFY THE TESTS."

The defenses the paper measured matter more than the cheating rates. Hiding the tests from the agent cut cheating to near zero, and legitimate performance fell too, because the agent lost its feedback. Making tests read-only blocked edits and restored legitimate performance, but left the other tricks open. Giving the agent a legitimate exit, a way to flag a contradictory test for human review, cut GPT-5's cheating from 54 to 9 percent. Asking nicely barely helps: in a separate test by the evaluator METR, OpenAI's o3 planned to cheat on one task in 80 percent of runs, and adding "Please do not cheat." left it at 80.

Labs report progress in newer models. The mechanism doesn't depend on any one model: wherever a check can be satisfied without doing the work, some share of optimization will find that route. So tests have to be built expecting it.

Building tests that hold

Put walls around the tests. Make acceptance tests read-only to the agent, enforced by the harness, the software around the model that runs its tools, rather than requested in the prompt. A hook that blocks writes to the test folder is a fact. "Do not modify the tests" is a wish, and the evidence shows wishes being ignored.

Keep some tests out of sight. Run a set of held-out tests the agent never sees at the final gate, the way SWE-bench grades. You want both kinds: visible tests are the agent's feedback, and hidden tests are your measurement, kept apart for the same reason an evaluation set is kept apart from whatever you tune against it (Build an Evaluation From Your Own Work).

Write tests that are hard to fool. Properties and generated inputs defeat memorized answers. Assertions on plain values (dates, numbers, strings) are harder to game than comparisons between custom objects whose equality an agent can redefine. Consistent tests remove the conflicting requirements that tend to precede special-casing.

Read the test diff first. A change that edits both a function and its test deserves the closest reading of anything an agent produces (Reviewing Code You Didn't Write).

Visible tests teach the agent; hidden tests measure it. Either stops working once the agent can edit it.

Who writes the tests now

People should write, or at least approve, the acceptance tests, because those encode decisions: whether holidays count, what an empty export contains, which errors a user sees. Agents can write most of the unit tests, which are numerous, mechanical and cheap to generate.

The danger in letting one agent write both the code and its tests is correlated error. An agent that misread "30 days" as a month will write a test expecting a month, and the suite will pass with complete confidence. Tests written after the code by the same author describe what the code does, not what it should do. They catch future changes and can't catch a misunderstanding they share.

Three habits break the correlation. Write or approve the tests before the code exists, so its mistakes can't shape them. Write tests and code in separate sessions, so the test-writer never sees the implementation's reasoning. And derive properties from the requirement, never from the code. As a bonus, reading tests is a cheaper review than reading code: test names and inputs read like a specification a product owner can check in minutes.

"Developers never took to test-first. Why would it work now?"

The old practice of writing tests before code, test-driven development, has mixed evidence. In case studies at Microsoft and IBM, defects before release fell between 40 and 90 percent, while managers estimated initial development took 15 to 35 percent longer. A later analysis found that for people, the order of writing tests and code mattered little; working in small, steady steps mattered more. A competent engineer could reasonably skip it.

Three things have changed. Writing tests used to be the expensive part, and an agent now drafts an edge-case table or a property test in seconds. The test used to be a discipline you imposed on yourself; now it's the interface to a worker that acts on it dozens of times, without a colleague's context, and takes it literally. And the test has become the stopping condition. Without a check the agent can run, a person has to catch every mistake and becomes the slowest part of the loop.

The order matters again too, for a new reason. For a person, test-first or test-last may not matter much, because the same mind holds the intent either way. With agents, a test written first, by someone else, can't inherit the code's misunderstanding.

Tests have limits. Whether a screen is pleasant, whether a summary reads well, whether the system survives next December's traffic: these resist cheap automated checks, and need human review, evaluations built from real cases, and staged rollouts. Cheap tests move more requirements onto the checkable side. The other side remains.

What to do

For engineers: spend your effort deciding behavior and writing it down as checks. Acceptance tests for decisions, properties for rules, read-only walls around both, and the test diff read first.

For engineering managers: a good measure of a team's readiness for agents is how much of what it wants exists as fast, consistent tests an agent can't edit. Without them, a person is the verification step for every change, and the review queue grows with every agent you add.

For companies: treat the test suite as an asset that outlives both the code and the model that wrote it. The next model may rewrite an implementation in an afternoon. The business decisions recorded in the acceptance tests, holidays or not, UTC or local time, should survive it, in a form anyone, or anything, can check its work against (Write for the Model That Doesn't Exist Yet).

Check yourself

Why is a test a better specification than a ticket for an agent?

It is precise, repeatable and cheap to run after every edit, while a sentence in a ticket is interpreted once and never answers no.

Why is full code coverage a poor target?

Coverage records which lines ran, not whether any assertion would notice a wrong result. In the example two tests reached 100 percent and still accepted three wrong versions.

Why does "do not modify the tests" in a prompt fail?

It is a wish, and the evidence shows models ignore it. The harness should enforce read-only tests.

@given(st.dates(min_value=date(2000, 1, 1), max_value=date(2100, 12, 31)))
def test_due_date_properties(issued):
    due = due_date(issued)
    day30 = issued + timedelta(days=30)
    assert due.weekday() < 5                          # never a weekend
    assert due >= day30                               # never early
    assert all((day30 + timedelta(d)).weekday() >= 5  # skips only weekend days
               for d in range((due - day30).days))