Measuring AI Impact Without Fooling Yourself

In February 2024 Klarna announced that its AI assistant had handled 2.3 million customer conversations in its first month, the work of 700 full-time agents. Resolution time had dropped from 11 minutes to under 2. The company projected a $40 million profit improvement for the year.

Fifteen months later, Bloomberg reported Klarna's chief executive saying that cost had been too predominant a factor in its evaluation, and that the result was lower quality. The company started recruiting human agents again.

Nobody needs to have lied for that to happen. The first numbers measured a step and a cost. The quality reading arrived more than a year later. And that sequence, fast flattering numbers first and the real answer much later, is how most companies measure AI today.

The fix is one comparison. The value of AI is the finished work you got, compared with what would have happened anyway. "Finished" means a customer, a colleague or a system accepted the work and used it; a draft, a prompt or one faster step doesn't count. "What would have happened anyway" is the counterfactual: nobody observes it, so you have to construct it. Most AI metrics in circulation fail one half of that comparison. Many fail both.

Why AI metrics flatter themselves

Every AI metric sits somewhere between the tool and the result. Near the tool: seats, logins, prompts per user. Then activity: drafts generated, lines of code, tickets touched. Then how much time people say they saved. Then the time one step takes, which usually overstates what the whole job gains (Where the Saved Time Goes). Further down, the ones that describe finished work: lead time from request to usable result, quality where the work lands, and cost per finished task. The closer to the tool, the faster a number moves and the easier it is to inflate.

AI adds a twist that older technology didn't. Activity metrics count output, and output is exactly what AI made nearly free. A count that can be pushed up at no cost will be pushed up. In 1975 the economist Charles Goodhart observed that a statistic tends to stop behaving once you lean on it for control; the popular version, Goodhart's law, says that when a measure becomes a target it stops being a good measure. Tell a team to raise AI usage and it will, by the cheapest route, which for most AI metrics is to generate more.

A metric that counts what AI makes cheap will always look good.

Self-reported time is no safer. In a 2025 randomized trial by the research group METR, experienced open-source developers expected AI to make them 24 percent faster. Afterwards they believed it had made them about 20 percent faster. Measured by the clock, they took 19 percent longer. One setting, one moment in the tools' history, but it shows how far feeling and stopwatch can drift apart.

And even an honest time saving isn't value yet. A large Danish study linked surveys of about 25,000 workers to employer and earnings records. Users reported saving under 3 percent of their hours; the records showed no detectable change in earnings or hours worked, and most users said they spent the saved time on other tasks. The savings weren't imaginary. They just never reached a ledger anyone was watching. A reported saving has to be followed into one (fewer hours paid for, more work shipped, more revenue) before it counts.

Compared with what?

Even a good metric needs a comparison, and the default one, before versus after, is contaminated in ways that make a useless tool look good.

Take regression to the mean, the tendency of unusually bad (or good) results to drift back toward average on their own. Teams often get picked for a pilot because they're struggling, and part of what makes a team struggle in a given quarter is bad luck. Suppose a company's teams average 6.5 days of lead time, the pilot team ran at 9.0 last quarter, and half of any team's deviation from average tends to persist from one quarter to the next while the other half is noise. With no tool at all, you'd expect that team at about 7.75 days next quarter. The pilot appears to cut lead time by 14 percent, and the tool did nothing.

Seasons do the same: a pilot launched in a slow month looks better than it deserves. So does novelty: people who've just been given a tool, trained on it and watched behave differently for a few weeks. So does everything else that changed at the same time, like a new knowledge base, a product release or a new manager.

The standard remedy is a comparison group measured over the same period. The simplest way to use one has a forbidding name, difference-in-differences, and a simple idea: subtract what happened to the group without the tool from what happened to the group with it.

Say both teams are measured on cases closed without reopening, per agent per week. Team A averaged 40 in months 1 to 3 and 47 in months 4 to 6: up 7, or 17.5 percent. Team B, with no tool, went from 39 to 42, up 3, because the season was picking up. If Team A would also have risen by 3 without the tool, the tool accounts for 4 cases a week, about 9 percent. Before-and-after overstated the effect by three-quarters.

Scroll sideways to see the whole figure

The method rests on one assumption, called parallel trends: without the tool, the two groups would have moved together. You can't test that for the months after the rollout, but you can look at the months before, and in this example both teams were rising by one case a month. The most common way to break it is choosing the pilot team for its enthusiasm or talent, so it was already on a steeper path. That's the reason to let a lottery, or the training schedule, decide who goes first.

Which points to the cheapest good experiment most companies already have. Rollouts happen in stages anyway, because training, security review and change management take time. If the order is random, or at least unrelated to how well each team is expected to do, every team that hasn't started yet is a comparison group for those that have. A large study of customer-support agents used exactly this structure. The late teams wait either way. The staged design just turns their wait into an answer that funding and headcount decisions can rest on.

Before and after tells you what changed. Only a comparison tells you what the tool changed.

Where the cost lands

AI-produced work that looks finished and isn't moves its cost to someone else. Researchers named it workslop: AI-generated content that looks good but lacks the substance to move the task forward. In their 2025 survey of US employees, 40 percent had received some in the previous month, at about two hours each to deal with.

The mechanism matters more than the numbers. The sender's time saving shows up in the sender's metrics. The receiver's time loss shows up in nobody's. So quality has to be measured where the work lands: the share of items the next stage accepts on first pass, how often work comes back, defects found downstream, minutes reviewers spend per item. Add one check that doesn't depend on anyone's incentive: pull random items made with and without the tool, strip the labels, and have someone score them blind against a fixed rubric (Build an Evaluation From Your Own Work).

Quality also decides the cost. Suppose a team attempts 100 tasks with a tool that costs $0.50 per attempt in computing. Reviewers spend 6 minutes on each at $40 an hour, and 20 attempts fail and get redone by hand in 45 minutes each. That's $50 of computing, $400 of review and $600 of rework: $10.50 per finished task, against $30 if people did all 100 by hand. The tool wins clearly. But computing, the number vendors quote, is under 5 percent of the bill (Cost per Task, Not Cost per Token). And if reviewers stopped catching failures, the cost would look lower while the failures surfaced later, on someone else's ledger.

"AI saves 30 percent of agent time"

Now take the claim you'll actually hear.

The data given can't answer it. Here's an illustrative setup. The department handles 10,000 tickets a month. Handle time falls from 12 minutes to 8.4, so agent time falls from 2,000 hours to 1,400. Agents cost $30 an hour, the tool $3,000 a month. On paper, 600 hours are freed, worth $18,000 a month.

What that's actually worth depends entirely on where those 600 hours went.

• They leave the budget. Attrition goes unfilled and quality holds. Net value: $15,000 a month after the tool's bill. Cost per ticket falls 25 percent, from $6.00 to $4.50.
• Demand grew and they absorbed it. Also worth $15,000, but only if you really would have hired 600 hours of agents otherwise.
• Half leave, half become slack. Net value $6,000.
• They leave, and quality slips. Reopened tickets rise from 8 to 14 percent, and team leads add spot-checks. Rework eats 184 of the 600 hours. Net value about $9,500.
• Nobody changes staffing. Same people, same hours. The department pays $3,000 a month for nothing, and cost per ticket rises to $6.30.

Scroll sideways to see the whole figure

Same 30 percent, results from plus $15,000 to minus $3,000. And none of the cases touches customers. The best one is worth $180,000 a year. At an assumed margin of $1,500 per customer per year, losing more than 120 customers to worse service turns it negative, and no agent-hours report will ever show that. That's the Klarna lesson in miniature.

The 30 percent describes a step. The value depends on where the freed hours went.

The 30 percent can even be wrong as a measurement. Tickets handled with the assistant may be the easy ones. Handle time may leave out after-call work. If agents are held to a handle-time target, some of the 30 percent may be tickets closed fast and reopened later.

Four measurements separate the cases, and the headline leaves all four out: paid hours or headcount (did anything leave the budget?), ticket volume and backlog (did demand absorb it?), reopen rate plus a blind quality sample (did quality hold?), and a comparison group (what would have happened anyway?).

The clock problem

Timing can flip the sign of a metric in both directions.

Google researchers showed the first direction in 2015. In short experiments, showing more search ads raised revenue. In experiments run for about three months, users who had been shown poorer ads became less likely to click on ads later. Google used that to cut mobile ad load by about half, and reported it was good for users and the business. The AI version: an assistant that closes tickets fast but teaches customers the channel doesn't solve anything looks excellent in month one.

The other direction: some work pays off after the first project, and a 90-day return will miss it. The evaluation set that makes the next deployment cheaper to check, the cleaned data, the written-down exceptions (When Work Starts to Compound). Writing those off for lacking an immediate return confuses a slow signal with no signal. But "it compounds" can't become an open-ended excuse, because costs compound too. Make it testable: name a leading indicator and a date, for example that the second team deploys in half the time the first one took, by the end of next quarter. If nothing moves by then, the claim failed, and saying so is the honest outcome.

What I'd do

My advice is to do most of the measurement work before the rollout, when it's cheap.

Choose the finished unit and write down the definitions: the metric, the window, the comparison. A measure chosen after the data arrives will find something.

Record a baseline for eight to twelve weeks, and add one metric the tool shouldn't affect. If it moves when your main metric moves, something else changed.

Let a lottery or the training schedule decide who goes first, so the teams still waiting become your comparison.

Measure quality where the work lands and cost per finished task. No tool dashboard collects either by default.

Report to the board by rung. "Handle time fell 30 percent on assisted tickets" is a step claim; label it as one. Give the comparison and the period, a range rather than a single number, and say whether each figure is measured, estimated or expected. Finance will want to know whether hours were released, cost was avoided, or cost was actually removed. Only the last changes the profit and loss statement without another decision.

Ask what the number will decide. If the program continues at any plausible value, extra precision isn't worth buying. Spend the effort on a quality guardrail instead.

Usage, activity and self-reported hours still belong on a dashboard, as signs that people are adopting the tool. Adoption is a precondition for value. It isn't evidence of it.

Check yourself

Why can't a before-and-after comparison show what a tool did?

Pilot teams are often picked when they are struggling, and seasons, novelty and other changes also move results. A comparison group measured over the same period lets you subtract what happened anyway.

Why can the same 30 percent cut in handle time be worth very different amounts?

The value depends on where the freed hours went. If nobody changes staffing, the department pays for the tool for nothing, and if quality slips, the savings can disappear.

Why measure quality where the work lands?

The sender's time saving shows up in the sender's metrics, but the receiver's time loss shows up in nobody's. Workslop moves the cost to someone else.