Confidently Wrong

In 2023 a New York lawyer had a doubt about a court decision ChatGPT had found for him, Varghese v. China Southern Airlines. So he asked ChatGPT whether the case was real. It said yes, and added that he could find it on Westlaw and LexisNexis.

The case did not exist. Neither did five others in the same filing. The judge in Mata v. Avianca called the legal analysis in one of the invented decisions "gibberish", and because the lawyers kept standing by the fake opinions after the court questioned them, he fined them and their firm $5,000.

The usual lesson drawn from that story is that models can't be trusted. The useful lesson is in the question the lawyer asked: he asked the process that produced the error to detect it.

Here is the idea this piece is built on. A confident wrong answer looks the same whatever caused it, but it can come from five different places, and each one is fixed somewhere else. Underneath all five, the way models are trained and scored pays them to guess rather than admit they don't know. The common word for these errors, hallucination, lumps the five together, which is why teams so often buy the fix for the wrong one.

Why a wrong answer sounds exactly like a right one

A language model writes by predicting the next token, a chunk of text a bit shorter than a word, over and over. During pretraining it learns to give high probability to whatever text actually came next in a vast pile of documents. Nothing in that objective says "be true". It says "be the likely continuation".

So the process that writes "Marie Curie was born in 1867" also writes a plausible wrong birth year for someone the model saw mentioned once, in the same confident register, learned from text where people state facts confidently.

A person who is unsure usually leaks it: a hesitation, a hedge, a change of tone. A model leaks nothing unless training taught it to, so the cue you rely on to judge an answer arrives whether the content is right or not.

Here's the surprising technical part. Inside, the raw model often does know, roughly, how sure it is. Calibration is the match between confidence and accuracy: a forecaster is well calibrated if it rains on about 70 percent of the days they say "70 percent chance of rain". OpenAI measured this for GPT-4 on multiple-choice questions, reading confidence from the probability the model gave each option. The pretrained model was almost perfectly calibrated, with an error of 0.007 on the standard measure. After post-training, the stage that turns a raw model into a helpful assistant, the error rose to 0.074. OpenAI's own report says post-training "hurts calibration significantly".

The raw result makes sense: pretraining penalizes overconfidence and underconfidence alike, so the model has every reason to keep its probabilities honest. Why post-training bends them is not settled; the likeliest explanation is that people, and the reward systems trained on their preferences, favor decisive, complete-sounding answers.

One caution keeps this from proving too much. Those probabilities are numbers researchers read off the model. A chat user never sees them, and a sentence like "I'm fairly confident" is just more generated text, with no guaranteed link to the numbers underneath.

The exam that pays for bluffing

Picture a multiple-choice exam with no penalty for wrong answers. A blank scores zero. A guess sometimes scores a point. Any sensible student fills in every bubble.

That is how nearly every influential AI benchmark is scored. In September 2025, Adam Tauman Kalai and colleagues at OpenAI and Georgia Tech reviewed ten widely used benchmarks and found that nine give no credit at all for "I don't know". Under that grading, they showed, abstaining is never the best strategy. Labs compete on those leaderboards, so models are tuned toward the behavior the leaderboards reward.

A small example shows how much this matters. Take 100 questions. A model knows the answers to 60. On the other 40, its best guess is right one time in four.

• The guesser answers everything: the 60 it knows plus a quarter of the 40. That's 70 right and 30 wrong.
• The abstainer answers the 60 it knows and says "I don't know" to the rest: 60 right, none wrong.

Score by accuracy and the guesser wins, 70 to 60. Charge one point per wrong answer and nothing for abstaining, and the guesser scores 70 minus 30, which is 40. The abstainer still scores 60. Same two behaviors, opposite ranking.

Scroll sideways to see the whole figure

The general rule is one line of arithmetic. If a wrong answer costs nothing, always answer. If it costs one point, answer only when you are more than 50 percent sure; if it costs three, only above 75 percent. Kalai and colleagues propose writing that threshold into benchmark instructions, so the behavior that wins is the honest one.

This isn't hypothetical. In OpenAI's own system card for GPT-5, two of its models got almost the same share of about 4,000 short factual questions right, answered from memory: 22 percent and 24 percent. One gave a wrong answer to 26 percent of the questions. The other gave a wrong answer to 75 percent, because it almost never declined. On an accuracy leaderboard, the second model wins. Charge a point per wrong answer and it scores −51 per hundred questions against the first model's −4.

A scoreboard that gives nothing for "I don't know" is paying the model to guess.

The same incentive reaches into training. Labs increasingly train with reinforcement learning on checkable answers: the model tries a problem many times and attempts that pass an automatic check are reinforced. A checker that marks only right or wrong is the bluffing exam again: an abstention earns the same zero as a wrong answer.

Five places a confident error comes from

The incentive explains why models guess, not every wrong answer. Sort confident errors by where they came from and five sources stand out, each with its own fix.

Scroll sideways to see the whole figure

1. It never learned the fact. Kalai's paper opens with a demonstration: asked for the title of Kalai's own PhD dissertation, three popular chat models each produced a specific title and year, and none had the right ones. That is a rare, arbitrary fact, like anything that changed after the model's training cutoff or was never public, such as your own price list. The fix is to put the fact in front of the model (search, a database lookup) or let it say it doesn't know. More thinking doesn't help, because reasoning can't recover a fact that isn't there.

2. The context was missing, stale or wrong. A model answers from two sources: what it learned in training and its context, the material placed in front of it for this request. When the context is wrong, a faithful model repeats the error with full confidence. In May 2024 Google's AI Overviews told users to add glue to pizza sauce, because the system had pulled a joke from a discussion forum. The model summarized what it was given; the retrieval failed. Quieter versions: last year's price list retrieved next to this year's, or nothing relevant found, so a model told to answer answers from memory and nobody can tell (What Fits Versus What Gets Used). A bigger model doesn't fix bad context. It summarizes it more fluently.

3. It reasoned badly over the right facts. Every line item on the invoice was correct and the total is still wrong. The contract says "unless" and the summary drops the exception. For exact operations, stop asking the model to do them in its head: a calculator or a few lines of code is exact. For multi-step logic, models trained to reason before answering make fewer of these errors (What Thinking Buys), and a separate check catches more. This is the one row where retrieval is useless, because the information was already there.

4. It followed the wrong instruction or format. Ask for three reasons when the source gives two and you will often get a third. Require a "publication date" field in a fixed output format and the model will fill it, even when the source has no date. Features that guarantee valid structured output, known as constrained decoding, make this sharper: they guarantee the format and say nothing about whether each field is true. Text inside a document that reads like a command can win too (prompt injection is the hostile version). The fix is in the request: make "unknown" a valid value, ask for "up to three", keep instructions apart from material.

5. It was rewarded for guessing. This one sits under the other four. A model that has learned that answering beats abstaining turns a knowledge gap or an empty search into a confident guess instead of a flagged gap. The fix is in how answers are scored, in training and in your own evaluations: count wrong answers separately from abstentions, and make a wrong answer cost more.

The same confident sentence can come from five different failures, and each one is fixed somewhere else.

Go back to the lawyer. His filing stacked several of these at once. A legal citation follows a rigid pattern, so a model can produce one that looks exactly right, while the specific numbers are arbitrary facts it can't reliably recall; the fake Varghese carried a docket number belonging to a different, real case. The chat product he used didn't search a legal database, so nothing anchored the output. He asked for supporting authority, so the model supplied it. Then he asked the same model to confirm, which meant asking the same gap and the same incentive to catch themselves. By October 2026 one public database listed more than 2,000 court decisions worldwide dealing with AI-invented content in filings.

Why "don't make things up" mostly fails

Telling a model "do not hallucinate" is the most common fix and the least effective. It adds no knowledge and no way to detect errors. For the first four sources, it asks the model to avoid an error it can't see.

It partly works in two situations, and both make sense once you separate the sources.

The first is when the instruction changes the scoring the model thinks it is under. Explicitly allowing "I don't know", as Anthropic's documentation recommends, or stating what a mistake costs, gives a model with some internal sense of its own knowledge permission to act on it.

The second is when the instruction turns a memory question into a lookup. "Answer only from the attached documents, quote the passage you rely on, and say 'not found' if nothing supports the answer" converts source 1 into source 2, which can be checked against text on the page. A quote either appears in the source or it doesn't.

The cost runs both ways. A model pushed to abstain too readily becomes useless, and the threshold that's right for a legal citation is wrong for brainstorming product names. Set it by what a wrong answer costs in the task.

"But newer models hardly hallucinate"

This is the strongest objection. On real ChatGPT traffic with browsing, OpenAI reported in August 2025 that GPT-5's thinking model had a 65 percent lower rate of incorrect claims than o3, and that 4.8 percent of its responses contained at least one major factual error, against 22 percent for o3.

My view: falling error rates move the question from whether to check to where to check. Three reasons.

The rate depends on the task. On a hard factual benchmark in October 2026, the frontier models gave a wrong answer to between 8 and 24 percent of questions, and the most accurate model was also the one that gave the most wrong answers. The work where you most need an AI answer (an unfamiliar regulation, a rare part number, a niche precedent) looks more like that benchmark than like a trivia quiz.

Progress isn't steady. On a benchmark of questions about people, OpenAI's o3 was wrong 33 percent of the time, double its predecessor o1's 16 percent, because it attempted more. A new model can be better on average and worse on your task.

Small rates at volume are large numbers. At 4.8 percent, a team asking 1,000 questions a day gets about 48 answers with a major error every day. And when nearly every answer is right, careful checking starts to feel like wasted effort and tends to lapse, which is my reason for keeping review wherever errors are costly.

What to do

When you read an AI answer, ask which source is likely. A specific fact from memory (a name, a number, a citation, a date) is source 1: verify it. A summary of your documents is sources 2 and 4: check they were the right documents and that nothing filled a gap. A calculation is source 3: recompute it in code. A confident answer where you'd have expected uncertainty is where guessing hides.

Resolve every citation with something other than the model that wrote it. Open the case in the database, fetch the link, search the source for the quoted string. That is the whole lesson of Mata.

Know what a spot-check proves. If you check 50 answers and find no errors, you can only say with 95 percent confidence that the true error rate is below about 6 percent. Showing it's below 1 percent takes about 300 clean checks. A handful of good results tells you almost nothing about a rare error, so keep a running evaluation set of real cases from your own work (Build an Evaluation From Your Own Work).

Score abstentions separately from wrong answers, in your evaluations and in how you judge vendors. A product that displays "I don't know" as a failure invites tuning that removes it. The most useful assistant behavior is marking a gap instead of filling it, and the most useful reader behavior is rewarding it.

For a team, watch for errors that spread. A plausible wrong figure in a shared document gets copied and built on, and nobody re-derives it because it looks finished; that is how workslop spreads. Five people asking the same assistant amount to a single source.

For a company, spend verification by source. Deterministic checks where exact answers exist (numbers, identifiers, code), retrieval with openable citations where facts change, and human review where an error becomes irreversible or public: the filing, the payment, the message to a customer. A second model reviewing the first is weaker than it looks, since models trained on similar data can share blind spots (Same Model, Different Results).

Choose the checks by the source of the error, and measure models by what a wrong answer costs you.

Accuracy alone pays the model to guess. Pick the metric that charges for the mistakes you actually can't afford.

Check yourself

Why was asking ChatGPT to confirm the case a weak check for the lawyer?

He asked the process that produced the error to detect it. The same knowledge gap and the same pull toward supplying an answer were still in play.

Why does a benchmark that gives nothing for "I don't know" produce guessing models?

With no penalty for a wrong answer, answering is never worse than abstaining. Labs compete on those scores, so models are tuned toward the behavior the scoreboard rewards.

Why does telling a model "do not hallucinate" mostly fail?

It adds no knowledge and no way to detect errors. It helps only when it allows "I don't know" or turns a memory question into a lookup in attached documents.