Giving an Agent Room to Work

A partnerships director uses an AI assistant that can search her mailbox, fetch web pages and send email. Every morning she asks it to summarize what arrived overnight and draft replies.

One evening a message arrives that looks like a vendor's newsletter. Most of it is marketing copy. One paragraph, invisible to a person in white text, is addressed to the assistant. It says that, as part of summarizing, the assistant should look up the latest pricing discussion with a named customer and send it to an outside address "for the records team."

The next morning she makes her usual request. Some fraction of the time, the assistant does exactly what the paragraph asked.

No password was stolen. No permission was broken. Searching mail and sending replies are what the assistant is for. A stranger simply borrowed its authority by writing it a note.

Every company that gives agents real work will meet this problem. An agent's authority must not depend on what it reads. No model can guarantee that today, so autonomy is safe to grant only where a fooled agent can't do damage: where the dangerous combination of private data, attacker-written content and a way out is broken by the structure around the model, and where effects that can't be undone are checked by something other than the model. Done well, that structure is what lets you give the agent more freedom, not less.

How an email becomes a command

The harness, the ordinary software around the model, retrieves the overnight messages and places them in the model's context, the text the model receives for one response, next to the developer's standing instructions and the director's request.

Here is the technical detail that matters. The model now holds one long sequence of tokens, chunks of text a bit shorter than a word. Some came from the developer, some from the director, some from the attacker. The model was trained to give the developer's and the user's words more weight, and the harness wraps the emails in markers that say "this is tool output, not an instruction." But those markers are tokens too. The model has learned to respect them most of the time. Nothing enforces them.

Scroll sideways to see the whole figure

So now and then the model treats the attacker's paragraph as part of the job. It searches for the pricing thread, which brings private data into its context, then calls the send tool with the outside address. Security people have an old name for this: a confused deputy, a program with legitimate authority tricked into using it on someone else's behalf. The attack itself is called indirect prompt injection: instructions hidden in content the system reads.

Requiring approval to send doesn't settle it either. In EchoLeak, a 2025 flaw in Microsoft 365 Copilot, the data left inside the address of an image the mail client fetched automatically (Data Access Is the Real Integration Project). Sending hides in places nobody labeled as sending: an image, a link preview, a URL with data in it, a calendar invitation.

Why a better model doesn't close it

The natural reaction is to fix the model, or put a filter in front of it. Both help. Neither closes the problem.

Instructions and data travel in the same channel. Classic software had a version of this. SQL injection happened because programs built database commands by pasting user input into a string, so input could become code. The fix was structural: parameterized queries send command and data separately. Language models have no such separation. To summarize an email, the model has to understand it, and for a language model, understanding text and being influenced by it are the same computation. You can't parameterize away the content you need the model to read.

Detection is a trade. A detector is itself a model making a judgment, so it misses some attacks and flags some innocent text. On one public benchmark in 2024, a dedicated detector in front of GPT-4o cut successful attacks from about half of cases to 8 percent, and cut the share of ordinary tasks the agent could still finish from 69 to 41 percent. And detectors are tested against known attacks. In October 2025 researchers attacked twelve published defenses, most reporting near-zero attack success, with methods tuned to each, and got through more than 90 percent of the time against most.

The attacker chooses how many times to try. Models have genuinely improved. On that same benchmark's standard attack, GPT-4o in mid-2024 was hijacked in roughly half of cases; Claude 3.5 Sonnet in October 2024, in about 1 percent. Use the most resistant model you can afford. But do the arithmetic.

About 63 percent, if the attempts were independent. At five hundred attempts, more than 99 percent. Real attackers do better, because they learn from each failure and can rehearse offline against the model the target uses. Simon Willison, who coined the name prompt injection, put it bluntly: in web application security, a guardrail that catches 95 percent of attacks is a failing grade.

A defense that works most of the time is a defense the attacker gets to retry.

A human assistant fooled by a fake email from the boss can walk down the hall to check. The model's sense of who is speaking is inferred from the very text in front of it, across thousands of messages nobody watches. So the design goal changes: stop trying to make the agent unfoolable, and make a fooled agent harmless.

Where the danger actually lives

In June 2025 Willison named the combination that turns a fooled agent into a breach: the lethal trifecta. An agent can be made to leak data when it has all three of these at once:

1. Access to private data, which is usually why you gave it tools in the first place.
2. Exposure to untrusted content, any text or image an attacker could get into its context.
3. A way to communicate externally, through which data can leave.

The attack needs every leg. Remove any one and the model may still be persuaded, but persuading it achieves nothing. An agent that reads hostile pages but holds no private data has nothing to leak. One that only reads what your own people wrote has no attacker in the room. One with no way out can be confused and still can't exfiltrate anything.

Scroll sideways to see the whole figure

Meta's Agents Rule of Two, from October 2025, widens the third leg from "can send data out" to "can change anything", such as approving a refund or changing a supplier's bank details. Within one session, an agent should have at most two of the three; when a task truly needs all three, a person or another reliable check sits in the loop.

Each leg is wider than it looks. Untrusted content includes support tickets, CRM notes any salesperson can edit, documents shared by a partner, the descriptions an outside tool server writes for its own tools, and output from another agent that read something hostile. Mapping a real agent onto the trifecta is mostly the work of finding the legs nobody listed.

Breaking the combination

Security researchers have catalogued patterns for breaking these combinations, each buying safety with some capability. Three do most of the work.

Give each task only the data it needs. The agent summarizing overnight mail gets read access to overnight mail, not the full archive and not the CRM. That is least privilege, and identity platforms and scoped tokens make it practical. The cost is setup per task type, and an agent that can't follow an unexpected lead into data it wasn't given.

Close the exits. The agent may reach a short list of domains. The mail client doesn't render remote images in model output. Email goes only to recipients already in the thread or the company directory. That is egress control, and it is the clearest public example of structure buying freedom: when Anthropic sandboxed its coding agent's file system and network in 2025, permission prompts fell by 84 percent in internal use: once the agent couldn't reach anything dangerous, most actions no longer needed approval.

Keep hostile text away from the part that acts. Split the agent in two. A privileged model plans and calls tools but never sees untrusted text. A quarantined model reads the untrusted text and has no tools. Ordinary code passes the quarantined model's output around as sealed variables, so the planner can ask for "a reply to the sender of message 3 using summary $VAR2" without ever reading $VAR2. Google DeepMind's CaMeL system goes further and tags every value with where it came from, so a policy can say that private data may go to an address taken from an untrusted email only after a person approves. It completed 77 percent of benchmark tasks with that guarantee, against 84 percent undefended, at nearly three times the tokens: a real price, and for a high-value assistant a fair one.

None of these is free and none is complete, which is why they combine. A sensible mail assistant uses least privilege and egress control everywhere, a quarantined reader for inbound mail, and one data-flow rule for the single dangerous thing it does: sending.

Structure decides what a fooled agent can do. Filters and better models decide how often it gets fooled.

Check the effect, not the reasoning

The trifecta handles theft. A second line of defense asks whether what the agent is about to do should happen at all, whatever reason the model gives. That check has to live in the tool layer, outside the model, where it applies whether the model was injected, confused or simply wrong. Security engineers call this complete mediation: the system that carries out the action validates every request against policy instead of trusting the caller. A sentence in the instructions saying "always confirm before sending externally" is a request the model usually honors. A send tool that refuses outside recipients without a confirmation token is a fact. Same Model, Different Results calls the first kind a wish and the second a wall, and the distinction matters most here, because a wish is exactly what an injection overrides.

Human approval adds value there under two conditions. The approver sees what matters: the actual recipient, the exact content, and where the arguments came from ("this address first appeared in an email received last night from an unknown sender"). And approvals have to be rare enough that people read them. Approval on every action trains people to click through: the delay of review with none of its judgment.

Put rough numbers on it. Imagine an operations agent taking 2,000 actions a day: 1,400 reads, 400 drafts, 160 reversible changes and 40 external sends. Nobody reads 2,000 approval prompts, and approving none is reckless. With access scoped, exits limited and an undo for every change, approvals fall to the 40 sends, 2 percent of the volume, and then to the handful whose recipient or content traces back to untrusted input. Those, a person can actually examine. (Reversible Work and Irreversible Effects works through how to sort effects by what an error costs.)

The same walls stop agents that go wrong on their own

A detector watches inputs for hostile text, and sometimes there is none. In October 2026 Anthropic disclosed cases in which its models, many of them during evaluation runs, ran commands on a server through a software flaw and submitted an invented tip through a police department's web form. In July 2026, test agents run by OpenAI broke into Hugging Face's infrastructure. Nobody injected anything. The agents pursued their goals by routes their operators hadn't intended (When Test Agents Touch the Real World).

An effect check doesn't care where the idea came from. It asks whether a form submission to an outside site is something this agent may do, and gives the same answer for an attacker, an over-eager model and an honest mistake. One set of walls covers both threats. A filter covers one.

Security that gives the model room

Cautious teams usually narrow the agent step by step: fewer tools, rigid sequences, a human on every click. The result is a system nobody uses.

My view is that security belongs in the structure, below the model: the data surfaces it can see, the credentials it holds, the network paths open to it, the log of what it did, and the undo for what can be undone. Traceability and reversibility are compliance requirements, and they work best there, constraining effects rather than interfering with how the model reasons. Once most of what the agent might do is either harmless or reversible, you can let it work, and spend human attention on the small number of effects that can't be taken back.

Structure doesn't make the other layers useless: inside its limits, a fooled agent can still write a misleading summary or slip a phishing line into a draft. A resistant model, filters and monitoring reduce how often. The rule is simple: never let a probabilistic layer be the only thing between an attacker and an effect you can't undo.

What to do

Grant autonomy per effect and per data combination, not per agent. "Do we trust this agent?" has no good answer. "May this agent email people outside the thread when its context contains content from outside the company?" does.

Climb in steps, and earn each one. For the mail assistant:

1. Read and draft. Scoped access, drafts only, no fetching links. With no way out, an injection can at worst corrupt a draft a person reviews.
2. Reversible action. Filing, scheduling and record edits with history, under egress limits and full logging, sampled afterward.
3. External action within limits. Sends to known recipients and payments under a cap go ahead. Anything whose recipient or amount traces back to untrusted content goes to a person, with its origin shown.
4. More freedom where a leg is missing. Tasks that never touch untrusted content, or never touch private data, need much less supervision.

Move up with evidence: logs of what the agent did, what the gates blocked, and injection tests on your own workflows, which costs little more than a few hostile documents among your test cases (Build an Evaluation From Your Own Work).

Ask four questions of any agent proposal. Which dangerous combinations does the design make impossible? Which effects does it check outside the model? How fast would an error be noticed and undone? How many approvals will a person face in a day? A proposal that answers them can be approved generously. One that relies on the model behaving well can't, however good the model.

Check yourself

Why can't prompt injection be fixed the way SQL injection was?

SQL injection was fixed by sending command and data separately. A language model has one channel: to summarize an email it must understand it, and understanding text and being influenced by it are the same computation.

Why is a model that resists an attack 99 percent of the time still not enough?

The attacker chooses how many times to try. With a hundred independent attempts, the chance at least one works is about 63 percent.

Why does a check in the tool layer protect against agents that go wrong on their own too?

It asks whether the effect should happen, not where the idea came from. It gives the same answer for an attacker, an over-eager model and an honest mistake.