From a Good Demo to a Working Department

The demo went well. An accounts-payable team in a mid-size company answers about 2,000 supplier emails a month from a shared mailbox: where is my payment, please send the remittance, the purchase order number doesn't match, here are our new bank details. A vendor shows a system that reads each email, works out what kind of request it is and drafts a reply using the supplier's invoice status. On 50 emails picked by the team lead, it gets 47 right. The slide says "about 95%".

Nobody is lying. And the number tells you almost nothing about what will happen in the mailbox.

Ask a different question about that demo: apart from the model, who was doing the work? Someone chose the 50 emails. Someone exported the data so the system could read it. Someone was standing there to smooth over any odd output. The audience wanted it to work. Someone judged the results by eye. And nothing in the world changed between the rehearsal and the showing.

A demo is an experiment run by its author, and the author is silently doing six jobs. A department removes the author. Every one of those jobs still has to be done by someone, and most of the work of deploying AI is finding them and giving each an owner who will still be there in a year.

The case is invented and its numbers are plausible assumptions. The pattern is real: it is how many good pilots become disappointing rollouts.

The cases nobody picked

Start with the 47 out of 50. Two things are wrong with treating it as 95 percent, and the second matters more.

The first is size. A score of 47 out of 50 is consistent with a true rate anywhere from about 84 to about 98 percent: on 2,000 emails, somewhere between roughly 40 and 320 errors a month.

The second is selection, and it only pushes one way. The person picking examples picks the cases the system is meant to handle: the common, clean ones. Real mail has a few request types that dominate, and then a long tail, the many rare types that each show up a dozen times a month and together add up to a lot. A statement of account where the answer sits in an attachment. A question in Portuguese. A thread forwarded four times with the real question buried in the fourth message. A lawyer's letter. An email meant for someone else entirely.

Suppose the mailbox holds about 40 distinguishable types. Under a plausible curve, the five most common cover just over half the mail, and the 25 rarest cover about a fifth. The demo's examples all came from the top five. If the system is right 97 percent of the time on those, 90 percent on the middle band and 70 percent on the tail, it is right on about 89 percent of the whole mailbox. That is about 215 errors a month, not 100. And the tail, a fifth of the mail, produces almost two-thirds of the errors.

Accuracy on chosen cases describes the head of the distribution. The department lives on all of it.

The tail is also where an error costs the most. A wrong answer about a routine payment costs a supplier a day. A drafted reply to a request to change bank details is another matter, because that request is the standard shape of payment fraud and belongs with a person who follows a callback procedure. The demo counted every error the same. The department can't.

Where an error can be seen

Every system makes errors, including the human one this replaces. What decides a rollout is where a person gets the chance to see them.

Take the common design. The system flags about a fifth of emails for a person (low confidence, an attachment it can't read, an unknown sender) and handles the other four-fifths on its own. A weekly audit reads a 5 percent sample of the handled ones. Now be generous and assume the demo's 95 percent really holds, so there are 100 errors a month. Assume flagging works reasonably well, so 60 of the errors land among the flagged emails and 40 among the handled ones, and that reviewers catch 85 percent of the errors they see.

Follow the 100 errors through.

Scroll sideways to see the whole figure

Fifty-one are caught in review. Nine get approved by a reviewer and sent anyway. Two turn up in the audit, after the supplier already has the reply. Thirty-eight sit in emails nobody read, and surface only when the supplier writes again, sometimes after acting on the wrong answer.

So 47 errors a month reach a supplier with nobody at the company aware of them. About two every working day. At the more realistic 89 percent, it's about 100.

Look at what made the difference. The 60 errors in flagged emails and the 40 in handled ones were equally wrong. One group was caught at 85 percent and the other at 5 percent. Improving the model a little moves the total. Changing where a person can see the work moves which errors get caught.

Where in the design a person can see an error matters more than how many errors there are.

It gets worse in a way that looks like good news. By week three, reviewers approve 91 percent of flagged drafts without changes. The dashboard celebrates. But if 15 percent of flagged drafts are wrong and reviewers approve 91 percent untouched, they can be catching at most about 60 percent of the wrong ones, not 85. Human-factors researchers call this automation bias: when an automated aid is usually right, people stop checking it properly, and the effect shows up in experts as well as novices and doesn't go away with training or instructions. At 60 percent, unnoticed errors rise from 47 to 62. Lisanne Bainbridge named the irony in 1983: automation leaves the operator the tasks its designers couldn't automate. The person on queue duty spends about 95 minutes a day on the hardest fifth of the mail, and her name is on every reply she approves.

The six jobs the author was doing

Go back to the six jobs and watch each one come due in the first month.

The case mix. The author chose the cases. The department receives all of them. That's the tail above.

Access and integration. The demo ran on an export and a spreadsheet. Production needs read access to the mailbox, a lookup into the ERP (the system of record, the one database the company treats as the truth about invoices and payments) and somewhere for drafts to go. Each is a request to a different owner. Mailbox access arrives on day 8 after a security review, the invoice lookup on day 15. In between, the system drafts replies without invoice status, writing "we are looking into your inquiry" where the demo said "payment is scheduled for the 14th". The team is running a different product from the one it approved (Data Access Is the Real Integration Project).

Exceptions and escalation. The author handled odd outputs on the spot. In the department, every type of email that needs a person (bank-detail changes, legal letters, disputes over amounts) needs a named route: who handles it, how long it may wait, what the supplier sees meanwhile.

Incentives and trust. The audience wanted the demo to work. The clerk on queue duty didn't ask for any of this, and her day just got harder before it got easier. When she says "the flagged ones take me as long as writing them myself", she is giving you a measurement of the design.

Measurement. The author judged by eye. At month end the dashboard shows 80 percent of emails handled automatically and 91 percent of drafts approved unedited. Both numbers measure activity. Neither says whether the department is better off, because nobody recorded a baseline, a measurement of how the work went before the change: how long handling took, how often the human replies were wrong. People make mistakes too. Without that count, the argument is between those who remember the system's worst blunder and those who remember the old backlog.

Count the minutes instead. Review, audit, repairs and upkeep take about half the time the team used to spend in the generous case, two-thirds at the realistic accuracy. Real savings, and far from the dashboard's 80 percent, which counts the emails nobody read.

Ownership and upkeep. The world held still during the demo. In the department, eleven suppliers are onboarded, two change names, and the company moves its weekly payment run from Thursday to Tuesday. Fourteen reply templates mention Thursday and nobody knows which. The system keeps promising Thursday. The model did nothing wrong. This is drift, the slow gap between what a system was set up to know and what is now true, and it happens whenever nobody owns the job of telling the system that something changed.

Scroll sideways to see the whole figure

Then a supplier phones: it was promised Thursday, paid Tuesday, and its own payments are now late. The vendor, IT, the team lead and the reviewers each explain why it isn't theirs. The supplier doesn't care. In 2024 a Canadian tribunal held Air Canada responsible for a fare rule its website chatbot got wrong, and dismissed the idea that the chatbot was somehow responsible for its own actions. The amount was small. The principle is the one every department should plan around: the customer deals with the company.

What a better model fixes, and what it doesn't

The obvious objection is that all of this is a capability problem, and next year's model will make it go away.

Some of it, yes. Accuracy on Portuguese emails and attachment tables is largely a question of how good the model is, and it will improve. But access, exception routes, incentives, the baseline and ownership are facts about the organization, and they stay put when the model improves. A better model doesn't grant itself permission to read the ERP, doesn't decide who handles a legal letter and doesn't notice that treasury moved the payment run. An upgrade can even add work, because it changes behavior across every template at once.

There's a simple way to tell which story you're in. Run the system on last month's real mail. If it is close to the demo's accuracy on the common types and the failures cluster in the tail, in permissions and in missing context, the gap is deployment work, and deployment work is what you should be doing. If it is well below the demo even on the common types, the gap is capability, and the right move is to wait or narrow the scope.

"Just iterate in production"

The second objection is the strongest, and it's half right. You can't know the long tail in advance. Analysis before launch has diminishing returns. A small start in the real flow with fast fixes usually beats months of planning, and my advice is never to spend those months.

But whether iterating live is safe depends on what kind of errors it produces. Three properties of a mistake decide it. Is it visible? Thirty-eight of the 100 errors only surfaced when a supplier wrote again. Is it reversible? A misfiled email can be moved; a promised payment date can't be unpromised, and a reply to a fraudulent bank-detail request can't be recalled. How far does it spread? An error that reaches one supplier group is cheaper than one that reaches all of them.

That gives you a better version of "iterate in production". Run the system in shadow mode: it reads live mail and produces its classification and draft, nothing leaves the building, and you compare its output with what your people actually did. A few weeks of that give you the real case mix, a baseline, an exception list and your first test cases, with no supplier exposed to anything. Then roll out in steps with a switch that turns it off (Moving Without Breaking covers the sequence).

Iterate in production, and choose which production.

There's a trust cost too. People lose confidence in a machine faster than in a colleague who makes the same mistake, so a bad first two weeks can leave a team that never agrees to expand.

What I'd do

None of this needs months. Most of it is a week or two, in this order.

Replay last month before launch. Run a month of the real mailbox, attachments and forwarded threads included, through the production path, with the people who did the work labeling what the right answer was. Don't sample at random, or the tail barely shows up; take about 100 cases from the common types, 100 from the middle and 130 from the tail. That's about eight hours of labeling, and it gives you a per-band error rate you can actually believe. Those labeled cases become your first evaluation set, a fixed collection of real cases with known right answers that you rerun whenever anything changes.

Name one owner. One person, with hours in their week for it, who can change the configuration and whom a supplier can reach. Plus a deputy for holidays. This is the job the demo's author was doing, and if nobody holds it, it doesn't happen.

Give every exception a route. From the replay, list the types that need a person and write down who handles each, how fast, and what the supplier sees in the meantime. Bank-detail changes go to a person with a callback procedure, always.

Measure the baseline in the same two weeks: volume, a timed sample of handling, time to first response, and an audit of a couple of hundred past human replies.

Check the checkers. Once live, mix a few known-bad drafts into the review queue and track how many get caught. A catch rate sliding toward 60 percent shows up there weeks before it shows up in supplier complaints. Rotate reviewers, and treat them as the first critics of the design. Their corrections are the best tail cases you'll ever get.

Build the way back first. Drafts are reviewed and not sent automatically for the first weeks, and one switch returns the whole mailbox to manual handling in minutes.

Write down "good enough to expand" before the first live day, while nobody is attached to the results. Something like: no severe error reaches a supplier for four straight weeks, unnoticed errors no more frequent than the human baseline, reviewers catching most of the planted bad drafts, faster first response, and the owner signs. Then expand in steps, the common types first and the tail last.

If the mailbox works, another department will ask for the same thing. The connector, the review screen and the replay habit carry over. The case mix, exceptions, incentives, baseline and owner don't, because they belong to the department. The second demo will look better than the first, since the builders have practiced, and that's exactly when skipping the replay is most tempting.

Check yourself

Why does 47 out of 50 not mean 95 percent on the real mailbox?

The cases were picked from the common, clean types. The rare types in the long tail are a fifth of the mail, handled less well, and produce almost two-thirds of the errors.

Why were the 60 errors in flagged emails and the 40 in handled emails so different in effect?

They were equally wrong, but reviewers caught 85 percent of the first group and the audit caught about 5 percent of the second. Where a person can see the work decides which errors get caught.

Why run the system in shadow mode first?

It reads live mail and produces drafts, but nothing leaves the building. In a few weeks you get the real case mix, a baseline and your first test cases without exposing any supplier.