Most of What a Company Knows Isn't Written Down

A water damage claim comes in from a ground-floor shop: $11,800, photos attached, contractor's estimate attached. The insurer's rule says claims under $15,000 with complete documents go on the fast track, paid without an inspection. The AI system reading the claim proposes exactly that.

A handler looks at the photos and sends an adjuster instead. In every room, there's a tide mark on the wall at the same height. That pattern means rising water, which this product excludes, and not wind-driven rain, which it covers.

That rule isn't written anywhere. It lives in the heads of the people who handle water claims.

Every company runs on knowledge like this: the customer who pays late every December and always pays, the plant manager whose "urgent" means this week while another's means today, the reason the payment run happens on Thursday. None of it is in the procedure, and all of it decides whether an automated system helps or does damage.

An AI system fills the gaps in what you wrote down with how an average company would handle the case. The usual fix, asking experts to write everything down, produces the official version of the job and misses the rest. What works is capturing one sentence of reason at the moment the knowledge shows up in the work, and the cheapest of those moments is when an expert overrules the machine.

The model answers for the generic company

A language model can work only with what reaches it: what it learned in training, and what the software around it places in its context, the text it reads for the task at hand. Your company's exceptions are in neither.

So the model does what it was trained to do. It produces the most plausible continuation, and the most plausible answer to a claim is the one a typical insurer would give. The result is an answer for an idealized version of your business: sensible, consistent with the written policy, and wrong exactly where your company departs from the average. Because it comes from a coherent picture, it sounds as confident as a right answer.

A better model doesn't fix this. It reasons more convincingly from the same incomplete picture. And the cost scales in a way people miss: a new hire who misses the tide mark gets corrected once, while an agent repeats the miss on every matching claim until someone notices.

Michael Polanyi put the underlying problem in a line in 1966: we can know more than we can tell. The name it gave rise to, tacit knowledge, lumps together very different things, though. Some knowledge lives in the body, like hearing that a bearing is failing. Some belongs to a group and is learned only by immersion, like what a long-standing client will tolerate. But much of what blocks an AI deployment in office work is plausibly a third kind, which the sociologist Harry Collins called relational tacit knowledge: knowledge that could be said and simply isn't, because the person who has it doesn't realize anyone lacks it.

That's encouraging. The exempt customer, the threshold nobody mentioned, the tide mark: all of it can be written down. The question is how to get it out.

Why "write down what you know" fails

The obvious approach is a documentation campaign. Ask your most experienced people to record what they know, put it in a wiki, give it to the model.

It works for the procedural parts of a job and has a long record of failing on the rest. The builders of expert systems in the 1980s hit this wall and called knowledge elicitation a major hurdle. Asked to explain themselves, experts produce a tidy rulebook, which is the part of their knowledge least in need of capture. Julian Orr, studying Xerox photocopier technicians in the 1990s, found that what let them fix difficult machines wasn't in the service manuals at all. It circulated in the stories they told each other.

A campaign fails three ways at once. Experts can't report what they don't know they know. Writing from memory, they reproduce the official version of the job. And the result sits in a wiki, months away from the case that would have tested it.

Capture at the moment it shows up

Tacit knowledge becomes visible at predictable moments in real work. Four do most of the work:

• An exception occurs. Someone departs from the procedure on a case in front of them.
• A decision is made. A threshold is set or a customer exempted, while the reasons are fresh.
• A mistake is corrected. A colleague or a customer catches an error, and the correction contains the missing rule.
• An AI output is rejected. An expert overrules a draft, a classification or a proposed action.

At each of these moments the expert is looking at the case and knows what was wrong with the default. Capture costs one sentence: what should have happened, and why. A month later in a workshop, the same person would reconstruct the case from memory and probably give the official answer.

Scroll sideways to see the whole figure

The fourth moment deserves special attention, because it's the cheapest to instrument: the rejection already happens inside software. And it turns the model's weakness into a tool. When a model applies the written procedure faithfully and an expert overrules it, the disagreement marks precisely a place where the procedure and the real work part ways.

An AI system's confident errors on your work are a map of what your company never wrote down.

The design work is attaching that sentence to an action people already take: a reject button that asks for a reason, a "departed from procedure" flag, a line in the change log for every threshold edit. Coding-agent vendors give the same advice for the instruction files their agents read at the start of each session. Anthropic's guidance says to add a line when the agent makes the same mistake a second time, when a review catches something it should have known, or when you find yourself typing a correction you typed last session.

What four weeks of disagreement buys

Here's how this plays out in numbers, for an illustrative claims team. The parameters are plausible assumptions, not data from any insurer.

Five handlers triage about 600 small claims a week. For four weeks an AI system proposes a triage for each claim without its proposals taking effect, a setup called shadow mode, while the handlers decide as usual. Whenever a handler disagrees with the system, she writes one sentence of reason. Half an hour a week, the team turns confirmed reasons into stored cases.

Assume 15 percent of claims, about 90 a week, turn on an unwritten exception like the tide mark. There are 150 distinct types of exception, and their frequencies fall off steeply: a few common ones and a long tail of rare ones, the rarest showing up about once every nine or ten weeks.

The team captures about 70 of the 150 types, fewer than half. But those 70 include nearly all the common ones, so they cover about 81 percent of the exception claims that arrive. The share of claims where system and handler disagree halves, from about 15 percent to 7.5. It costs roughly ten one-sentence notes a day across the team, each written while the handler was already looking at the claim.

Scroll sideways to see the whole figure

Three features should carry over to real teams even where the numbers don't.

The common cases come fast, the tail comes slowly. Week one alone covers 59 percent of exception claims. By week twelve the team has 94 percent of claims but only 77 percent of types. The tail takes months.

After the first month, most of what's left isn't missing knowledge. At week four, less than a third of the remaining disagreement is uncaptured exceptions. The rest is ordinary errors capture can't fix, and captured cases the system still mishandles because a stored case has to be found and applied. The work shifts from capturing to checking that captured cases actually change the system's behavior.

Watching misses the rarest cases entirely. About 56 of the 150 types never occur in four weeks. Some of those, like a pattern that only appears in storm season, matter most when they come. For those you need deliberate methods: interviews that walk an expert through one specific hard case with probing questions, which is called the critical decision method, or a pre-mortem, where the team imagines the system has caused a serious problem six months from now and writes down what happened.

What a captured case looks like

Format matters, because the record has to work for a model and for a newcomer. Where a real rule exists, write the rule, with its reason. Where the decision rests on judgment, a few worked examples carry it better than any rule, for an apprentice and for a model alike. Every format carries its reason, because a rule without one gets applied by its letter.

The tide-mark case, captured during shadow review, might read:

Case W2-041 · captured during shadow review · confirmed by team lead

Claim:      water damage, ground-floor shop, $11,800, photos and
            contractor estimate attached
AI proposed: fast-track payment (under $15,000, documents complete)
Expert:     send an adjuster
Why:        interior photos show a tide mark at the same height in
            every room. That pattern points to rising water, which this
            product excludes, not wind-driven rain, which it covers.
Applies:    water claims where photos show a level line across walls
Not when:   ceiling stains; damage limited to one exterior wall
Owner:      claims team lead · revisit if the product adds flood cover

The "Applies" and "Not when" lines let a model handle a case the record didn't anticipate. The owner and the revisit condition say who may change it and when it goes stale.

A model can help write these. It can run the interview, or read a quarter's override log and draft candidate rules. But a reason drafted from traces is the most plausible reason, which is the generic company's reason again. The draft is a hypothesis until the expert confirms it.

A model can draft the reason in seconds. The expert's correction of the draft is what makes it knowledge.

"Models will learn this from data eventually"

The strongest objection treats all this as a temporary chore. Models keep getting better at learning tacit patterns from examples, and companies are full of examples. Why ask experts for reasons?

Part of this is right. The best evidence is a large customer-support study: an assistant trained on past conversations raised issues resolved per hour by 15 percent on average and 36 percent for the lowest-skilled fifth of agents. Tacit practice that leaves a rich trace, with outcomes attached, can be learned from that trace.

It fails on four points, though.

Your exceptions aren't in anyone's training data. The tide-mark rule for this product exists only in your records, if anywhere. Learning it requires the data, labeled with outcomes, which is capture under another name.

Traces record what was done, not why. A model trained on past decisions learns past mistakes along with good judgment, and can't tell an exception from an error unless someone says which is which.

The rarest cases are too rare to learn from. More than a third of the exception types in the example never occurred in a month. Learning from data faces the same long tail as watching.

Policy changes faster than history. The day the product adds flood cover, every past tide-mark decision teaches the wrong rule. A captured reason with a revisit condition can be retired in a minute. A pattern learned from two years of decisions has to be unlearned.

So the objection ends up supporting capture. The support study was possible only because conversations with outcomes had been recorded at scale.

The people who hold it

All of this depends on experts writing that sentence, and they have reasons not to. Capture after the work loses to the queue, so the reason field has to sit inside an action they already take. The person everyone asks is often valued for exactly that; put her name on every case she supplies and make her the one who confirms the drafts, so she stays the authority. And capturing someone's judgment so a system can apply it looks like training a replacement, which sometimes it is. A company that asks for capture without saying who benefits should expect polite non-compliance (Why People Resist AI at Work, and When They're Right).

There's a slower risk too. Tacit knowledge is produced by doing the work. Handlers learn the tide-mark pattern by seeing hundreds of water claims. When a system handles the routine cases, people see fewer of them and stop accumulating that knowledge (Why the Process Exists). A successful capture practice can consume its own source. Teams that rely on expert judgment need deliberate exposure to ordinary cases, through rotation or regular review of a sample, or in a few years nobody will be able to correct the system.

What to do

Put a reason field on every override. Every place an expert can reject, edit or redo an AI output should ask for one sentence of why. That single change produces more usable knowledge than any documentation campaign.

Run new AI systems in shadow mode first, and treat the disagreements as the deliverable. The list of reasons is worth more than the accuracy number.

Turn confirmed reasons into cases with an owner and a revisit date, and turn rejected outputs into test cases the system has to pass on every change (Build an Evaluation From Your Own Work).

Go looking for the rare, expensive exceptions on purpose. One interview about the case your best expert would least trust a machine with, and one pre-mortem before launch.

Keep a person where the knowledge can't be captured. For judgment that resists every format, the record is a route: who decides, how fast, and a log of what they decided and why. Sometimes, after twenty logged cases, the judgment becomes stateable.

My advice is to start with the first one this week. It costs almost nothing, and every model you use from now on will be better for it (Write for the Model That Doesn't Exist Yet).

Check yourself

Why does a better model not fix the tide-mark problem?

It reasons more convincingly from the same incomplete picture. The exception is not in its training or its context, so it still answers for the generic company, with confidence.

Why does capturing a reason at the moment of an override beat a documentation campaign?

The expert is looking at the case and knows what was wrong with the default. A month later in a workshop, the same person would reconstruct the case from memory and probably give the official answer.

Why won't models simply learn these exceptions from past data?

Traces record what was done, not why, so a model can't tell an exception from an error. The rarest cases are too rare to learn from, and policy changes faster than history.