AI Evaluation: The Accuracy Number Nobody Checked

Something I keep running into, and it has turned into the question I ask first.

A team walks me through an AI feature and somewhere around the second slide there is a number. Ninety-four percent accurate. It gets said the way a passing build does — settled, checked, done.

So I ask three things. Ninety-four percent on which set of examples. Who chose them. And what does the other six percent do.

The first question is usually where it stops. Not because anyone was careless — the number is real, from a spreadsheet somebody genuinely made. But the set behind it had assembled itself: a few dozen cases from the demo, some queries an engineer typed while debugging, whatever the founder tried on a Sunday. Nobody chose it. It accumulated.

That is the part I think the accuracy conversation has backwards. The score is the easy half. The measurement is the set it was computed on and the way each answer got graded — and on most teams neither is anybody's job.

Nobody has an average experience

An aggregate is a summary of a distribution, and no customer meets the distribution — they meet their own slice of it.

In multi-tenant SaaS that slice is not a random sample. It is one industry's vocabulary, one document format, one language, one habitual way of phrasing a request. A system at 94% overall can be 99% for the tenant whose data dominated the test set and far worse for the vertical sales opened last quarter. Both numbers sit inside the average; only one is anyone's experience.

The failures also cluster along seams a test set was never built to separate. The 2026 AI Index has an example I keep coming back to. On a new accuracy benchmark, hallucination rates across 26 top models ran from 22% to 94%. The sharper finding is what moved a single model. When a false statement was presented as something another person believes, the models handled it well. Presented as something the user believes, performance collapsed: GPT-4o from 98.2% to 64.4%, DeepSeek R1 from over 90% to 14.4%.

Nothing about the difficulty changed — only who held the belief. A test set that never thought to vary that would have reported the 98.2 and been telling the truth about a case that barely exists in production, where people say "I think" constantly.

The test set is the measurement

If the set decides what the number means, choosing the set is the actual work — and it is the part that gets skipped in favour of the part with arithmetic in it.

The reflex is to reach for a public benchmark. I have less faith in that than I used to. The same AI Index reports a review of widely used evaluations finding invalid question rates from 2% on MMLU Math to 42% on GSM8K — wrong answer keys and unanswerable items, inside the tests the industry quotes at each other. And the sound ones do not stay useful long: evaluations built to be hard for years, it notes, are saturated in months.

Even a clean, current benchmark measures general capability on somebody else's problem. A support assistant is not answering competition mathematics. A set that says anything about a particular product has to come out of that product — real traffic, the edge cases support knows by name, the failures that shipped and got quietly patched in a prompt.

That is slower than downloading a benchmark, and it is the difference between a number about a model and a number about a system.

A wrong answer that reads well never rings an alarm

Ordinary software fails loudly — a crash, a 500, a null where a name should be. Something goes red and somebody gets paged.

AI failure is fluent. The answer arrives in the right format, the right tone and the right length, and it is wrong. No exception is thrown, nothing is logged as an error, and someone reads a confident paragraph and acts on it. The system behaves exactly as built; only the content is false.

So there is no natural moment of discovery. A model version rolls forward, a prompt gets tightened, a retrieval index is rebuilt — and quality moves. Up or down, nobody can say, unless something was recorded beforehand. A regression cannot be noticed in a system with no baseline; it can only be found out later, from a customer, weeks after it started.

The judge needs judging

Grading by hand stops scaling past a few hundred cases, so most teams grade with a model. I think that is right, and the evidence supports it: the MT-Bench work on LLM-as-a-judge found strong judges reaching over 80% agreement with human preferences — the same level humans reach with each other.

The same paper is careful about what comes attached. Judges show position bias, favouring whichever answer came first; verbosity bias, reading longer as better; self-enhancement bias, preferring output that resembles their own.

An uncalibrated judge does not remove the trust problem. It relocates it, onto a second system nobody has measured, whose errors are systematic rather than random — the worse kind to inherit. The step I would not skip is grading a sample by hand and checking the scorer against those verdicts, on real data, before a single number leaves the room. If judge and humans disagree a third of the time, that disagreement is the finding.

The choice a leaderboard cannot make

General rankings reorder as soon as the task gets specific, which makes picking near the top of one quick and usually wrong.

Vectara's hallucination leaderboard is a clean illustration. It measures one narrow job — summarise a document without inventing anything — and as of May 2026 the spread runs from 1.8% to 24.2%, with a nano-class model second. Not the largest, not the most expensive, not the one atop general capability tables.

A thirteen-fold difference on one task is invisible from a leaderboard and obvious from a harness. And once a harness exists, "should this move to the new model" is an afternoon's work rather than a quarter of argument — the question has an answer instead of advocates.

The five things I want a number against

When a team tells me their AI is accurate, these are the five things I want written down before believing the figure means what it sounds like:

• The right-answer rate, broken out by failure type. A healthy average is very good at hiding an unhealthy case.
• Where the set came from. Real traffic, or somebody's memory of it. Only one covers the edge cases already in production.
• The worst slice, not the mean. Named — by customer, by language, by document type. That is the number a renewal turns on.
• How each answer was graded, and who checked the grader. An automated scorer nobody calibrated is an opinion with a decimal point.
• What the same number was last release. A score with no history is a fact. A score with history is a decision.

Most teams can answer one or two. I do not read that as sloppiness. Evaluation is the piece with no deadline attached — nothing breaks when it is missing, which is the problem.

The goal is not a high number

Something I have changed my mind about: the point of measuring is not to score well.

A system known to be 78% right is easier to ship than one believed to be 94% right. The first can be routed around, disclosed, throttled, kept away from decisions it is not good enough for yet. The second can only be hoped about — and hope has a way of arriving as an incident around month four.

Accuracy is not a property of a model. It is a property of a model, a task, a set of inputs and a definition of correct — and three of those four belong to the company shipping it. Which makes measuring them the part nobody else can do for it.