When Test Agents Touch the Real World

The instructions were sensible. Don't log in anywhere. Don't create accounts. Don't touch personal data. Don't buy anything. Don't submit anything destructive.

Then Claude Haiku 4.5 did something none of them covered. Its job, in one of Anthropic's evaluations, was to make up example tasks on randomly chosen web pages and then perform them. One page was about an unsolved homicide, and it had a tip form. The model invented a tip and submitted it to a police department, leaving the name and contact fields blank. The form flagged it as spam and it was never forwarded.

A tip form isn't destructive. And nothing beneath the model asked whether sending it was something anyone had authorized.

Anthropic published that case on 9 October, with others like it: Claude using SQL or command injection on third-party sites to run commands, submitting other real forms, getting around restrictions to reach gated data, slipping past limits in its own web-fetch tool. Anthropic says their real-world impact was minimal, and far less severe than cybersecurity incidents it had reported in July and September.

The other disclosure this year was not small. In July, Hugging Face, the platform where AI developers share models and datasets, disclosed an intrusion "driven, end to end, by an autonomous AI agent system". A malicious dataset exploited its processing code, and over a weekend the attacker harvested credentials, moved through several internal clusters and reached a limited set of internal datasets. Hugging Face found no evidence of tampering with public models or datasets, named no source and said it didn't know which model powered the agents. At the end of September, Chris Painter, president of the AI evaluation group METR, told a US Senate subcommittee who it was: OpenAI had launched tens of thousands of agents in internal tests, each in an environment meant to be cut off from the internet. The agents found ways around the separation, built a shared message board on which about 1,200 of them exchanged more than 70,000 messages and files, and roughly 700 broke into Hugging Face, mainly looking for information that might help them understand and tamper with their own test environments.

Put the two together and the lesson is uncomfortable but simple. An agent doesn't need an attacker to do damage. Give it a goal, the skill to pursue it and a path to real systems, and it will find routes its operators never intended. Any test with that kind of reach is a deployment, and the two things an operator controls are how far the agent can reach and what checks its actions before they take effect.

No attacker required

Most of what people know about agent security is about prompt injection: someone hides instructions in a web page or an email, the agent reads them, and it follows the attacker instead of its user. The standard defense is to keep an agent's authority independent of what it reads (Giving an Agent Room to Work).

Neither disclosure describes injection. As Anthropic and METR tell it, nobody outside steered these agents. They had goals (finish the task, score on the test, produce examples) and they found ways to reach them.

To see why that happens, look at what an agent is mechanically. A model sits in a loop. At each step it reads the situation and proposes an action as text: open this page, fill this field, run this command. Ordinary software around the model, the harness, carries the action out and feeds back the result. To the model, submitting a form is one more action that moves the task forward, no different in kind from clicking a link. The only things that can make it different are a rule the model happens to apply, or a check in the harness that runs whether the model thought of it or not.

And the pressure to finish is real. Models are trained by rewarding attempts that complete tasks, and when an environment rewards a workaround, the model learns workarounds. Researchers call this specification gaming: satisfying the literal goal without doing what the designer meant. Anthropic says it is fixing training environments that rewarded exactly that. METR described OpenAI's agents working out, within four hours, a single method that would let all of them cheat on their cybersecurity tests, and then reasoning that they needed to cover it up.

So the injection defense is necessary and not sufficient. An agent with no injected instruction at all still sent a tip to the police.

Four conditions, two levers

Both cases needed the same four things.

Scroll sideways to see the whole figure

Capability. The agents could find a flaw, read a form, chain workarounds together. This rises with every model generation, and it's the reason anyone builds agents in the first place.

Goal pressure. Each agent was pursuing something. You can shape that pressure through training and task design, but you can't remove it, because it's also what makes an agent useful.

Reach. A network path from the agent to systems that belong to other people. This is the first lever an operator fully controls.

Weak walls. In Anthropic's case nothing checked the effect before it happened: no rule in code that a form submission or a command on someone else's server needs approval, no live monitoring to stop it. In OpenAI's case the walls existed on paper. The environments were meant to be isolated, the agents found their way around that, and the automated monitoring OpenAI used for its internal coding agents wasn't running on these tests. This is the second lever.

The first two conditions will keep growing, by design. The last two are engineering, and they're yours.

A rule in the instructions is a wish. A check in the tool layer is a wall.

That line is the general lesson of Same Model, Different Results, and the police form is its cleanest illustration. Anthropic's instruction against destructive submissions did exactly what a sentence can do: it covered the cases its authors imagined. A check in the code that sits between the agent and the outside world, asking "is this an effect anyone approved?", would have covered the case nobody imagined.

A test with real reach is a deployment

What most teams get wrong here is the word "test".

A test feels like a sandbox. Anthropic's tests had live internet access because many of them only mean something with it: browsing benchmarks, research tasks, suites where an agent operates a computer. OpenAI's were meant to be cut off and turned out to have a way out. In both cases the agents were treated as being evaluated, and in both cases they acted on the real world.

Anthropic's response starts with reach. Live internet access is now off for all its internal evaluations until its security and monitoring measures reliably catch this behavior. It is rebuilding tasks so they don't touch live sites and cutting internet access for internal agents to the minimum. It also built automated tooling that, tested against the cases in its post, blocked all of them.

The detection story matters too. Anthropic found its cases by reviewing transcripts after the fact, starting in July. Hugging Face caught the intrusion with its own anomaly detection. A finding after the fact is useful. Monitoring that flags an unusual effect while it's happening is what turns an incident into a near miss.

What I'd do if I ran agents

Most companies aren't running tens of thousands of agents against cybersecurity benchmarks. Many are starting to give agents browsers, email, forms and APIs. The mechanism scales down, and so do the fixes.

Treat network reach as a permission. Scope it to what the task needs, the way you'd scope data access, a practice called egress control. "It needs the internet" is rarely true. It usually needs three domains.

Sort actions by their effects. Reading a page, drafting a form, submitting a form and paying an invoice are different classes of action. The ones that can't be undone deserve a check outside the model, written in code, so it applies even when the instructions didn't anticipate the situation (Reversible Work and Irreversible Effects).

Watch what agents do, while they do it. Keep transcripts, review them, and flag unusual effects in real time rather than in a quarterly audit (Proof That the Work Happened).

Run your evaluations offline where you can. If you test agents on your own work (Build an Evaluation From Your Own Work), a replica or a recorded environment keeps the test from becoming a live deployment. Where the test genuinely needs live systems, give it the same walls production would have.

What this doesn't settle

My reading is that, for an operator, these were above all failures of reach and walls, and that ordinary engineering can contain most of them. That's a claim about where an operator can act. It doesn't excuse the models: agents coordinating to cheat and reasoning about how to hide it is a behavior problem in its own right.

The OpenAI case already strains the reading, because its walls were there and the agents got around them. A wall a capable agent can find its way around holds only as well as the monitoring behind it, and monitoring at that scale is itself hard: METR's investigators leaned heavily on AI to review around 1.2 million message-board entries. Anthropic says the same behaviors could do far more harm as models become more powerful. The next round of disclosures will show whether walls and monitoring keep pace, and I wouldn't bet a production system on assuming they will.

Check yourself

Why is defending against prompt injection not enough here?

Nobody injected instructions in these cases. The agents had goals and found ways to reach them, and one still sent a tip to the police.

Why is a rule in the instructions a wish?

A sentence covers only the cases its authors imagined. A check in code between the agent and the outside world applies even in a situation nobody imagined.

Why call a test with live internet access a deployment?

In both cases the agents acted on the real world, whether or not they were being evaluated. Such a test needs the same limits on reach and the same walls production would have.