Reliable, and still wrong: testing AI like a measuring instrument

Here is how most teams check an AI system, when they check it at all. Run the prompt. Run it again. Same answer both times, ship it. The check feels rigorous because it is repeatable, and it measures exactly one thing: whether the system is consistent. In measurement language that is reliability, and reliability is the property a stopped clock has in abundance. Perfectly consistent, right twice a day by accident, and wrong the rest of the time with total confidence.

Our last two articles argued about placement: first that the human gate belongs where a wrong-but-confident output would cause harm before anyone catches it, then that inside any workflow the harm concentrates in composition tasks rather than cued lookups. This article extends the thesis one layer further, into measurement: how you decide whether an AI output deserved trust in the first place. Like the last one, it rests on a small piece of direct evidence, a controlled test our founder ran during his graduate study at Purdue, on the exact task our own rental operation runs every day. The result splits cleanly down the middle of a single output, and the split is the lesson.

When someone says an AI system works, they are compressing three separate questions into one word. Reliability: same input, same machinery, same answer? Validity: is the answer right against a reference you trust? Replication: could someone else redo your procedure from your notes and expect what you got? The three are independent. A system can pass any of them while failing the others.

The pair that gets confused is the first two, and the stopped clock is the confusion made visible. Consistency checks can only tell you the clock has not moved. They cannot tell you the time. For that you need a trusted reference, and the reference has one non-negotiable property: it must exist before the run. Score a model against a key you write afterward and the key drifts toward whatever the model said. Score it against your memory of the right answer and the same drift happens, just more slowly. A reference is only a reference if it was locked while the machine could not yet influence it.

Replication is the quiet third question. It is what turns a private impression into a procedure: the exact prompt, the exact model version, the date, the saved transcripts. Without it, 'we tested it' means 'trust us.'

The test was built as a measurement exercise on the workload we know best. Ten guest messages from a short-term-rental operation, synthetic and de-identified, written to mirror real message traffic with no real guest data in them. The task handed to the model was deliberately half closed and half open: sort each message into exactly one of six operational categories, check-in and access, cleanliness and housekeeping, maintenance, billing and fees, stay changes, feedback and praise, then name the top three recurring themes across the set, ranked by how many messages touch each.

Before any run, the answer key was written and locked: the correct category for all ten messages, plus the operator conventions that settle the hard cases. A message goes to the category of its primary actionable ask. Anything that gates physical entry, door codes, lockboxes, gate codes, is access. Money already charged is billing, whatever the charge was for. Three of the ten messages were deliberate borderlines, built so those conventions would have to do real work. And one trap was set on purpose in the open half: the messages were composed so that billing issues and access failures would each touch exactly three of them. A tie, waiting to see what the model would do with it.

The protocol: the identical prompt, verbatim, three times, each in a brand-new session with no tools, no memory, and no sight of the other runs. The model was Claude, the same model family our own operation runs on, pinned by exact version ID rather than a hosted alias, because aliases move over time and a reliability claim is empty if the machinery changed between runs. All three runs happened on the same day, and every transcript was saved verbatim.

The sorting task did not wobble at all. All three runs assigned the same category to all ten messages, and every one of those labels matched the key: thirty for thirty, the three deliberate borderlines included. Better, the reasons matched the coding conventions and not just the label column. The message that bundled a pet-fee refund request with a broken gate code went to billing in every run, with the refund read as the primary ask, which is exactly the convention the key wrote down. The jammed lockbox went to access rather than maintenance in every run, the call the key's entry-gating convention dictates. The towel request stayed subordinate to the unchanged sheets it arrived with.

That half of the output earned its trust the only way trust is earned in measurement: reliable across independent runs, and valid against a reference that existed before the machine did anything. The stakes on that half are also low and reversible; a misrouted message costs a few minutes to re-file. Closed task, answer-keyed evaluation, cheap failure: that is the shape of work an operator can hand to a model with a clear conscience.

The theme ranking was a different animal. All three runs named the same three themes, billing, access, cleanliness, and nothing else, so at the level of what made the list, the synthesis held. The order did not. Runs one and three put billing first. Run two put access first, calling faulty entry systems 'the most common operational failure this week.' The label on the third theme drifted too, from 'Cleanliness lapses' in one run to 'Housekeeping gaps at turnover' in the other two.

Then the detail that makes this an article. By the model's own counts, billing touched three messages and access touched three. The trap had sprung: the top two themes were tied, exactly as designed. No run said so. Not one output contained a word about a tie, a tossup, or a judgment call. Each run asserted its order as if the data settled it, and each was fluent enough that a reader would never suspect the data had not.

Under the key's own scoring rule, both orderings are defensible, so the flip is not a validity error, and we want to be precise about that. It is something quieter and, for an operator, worse: the model was handed a coin flip, and it called the flip confidently, three times, without ever disclosing there was a coin. A consistency check cannot surface that. Neither can a single run. It took the pre-written key and three transcripts side by side to see it at all.

Readers of the last article will recognize the shape. There, on a synthetic operations log, cued retrieval was near-perfect and every verified error appeared in open composition. Here the same boundary shows up inside a single output: the closed half, sorting against fixed categories, was reliable and valid together, and the wobble lived exactly where the task went open-ended, in a ranked synthesis where the counting rule ran out and something had to break the tie. The pattern repeats across both tests: the closed part of a job earns trust through answer-keyed evaluation, and the open part is where models make decisions they do not tell you they made.

The stopped clock completes the argument. Three identical label sets prove the tool is steady, and that is all they prove. What proves it correct is the key written before the first run. Had the key been wrong, three agreeing runs would not have caught it; they would have marched together, in perfect agreement, off the same cliff. The mirror image matters just as much. Had we checked correctness on one run only, the silent tie-breaking would never have surfaced, because any single ordering looks reasonable on its own.

None of this is abstract for us. Message triage is precisely what our own agents do all day inside our rental operation: triage the guest messages and incidents, work the volume, and leave the exceptions for people to review. The closed half of this test is that workload in miniature, and results like these are part of why we are comfortable with the arrangement. But the ranking half is what an operator acts on. A theme ranking decides where the week's money and attention go. That is exactly the kind of output we route through a person, and this test is the measurement-layer justification for the routing.

The method transfers to any operation, and it is cheap. Write the answer key first, conventions included, before the model sees anything, so the reference cannot drift toward the output. Run the evaluation in fresh sessions, so no run can lean on another. Pin the model by exact version ID, not an alias, so the machinery is the same machinery each time. Date everything and keep the verbatim transcripts, so someone outside your team could redo the procedure and check you. None of this needs a platform or a budget. It needs discipline and an afternoon.

Then draw the trust boundary the results draw for you. Let the model own the closed, answer-keyed, cheap-to-reverse work. Put a human gate on anything ranked, because rankings are where ties get broken silently, and on anything that touches money, a refund, a fee decision, before it executes. This is the measurement layer of the discipline our frameworks already impose on process. ESSA will not let technology touch a process that has not been standardized, and the maturity model will not let automation in before Level 3. Same logic, one layer up: no AI output graduates past its human gate on the strength of a consistency check alone.

The honest limits, as usual. This was one task, ten synthetic messages, one pinned model, three runs. Consistency observed across three runs is observed consistency, not a mathematical property; sampling variance means even conservative settings do not guarantee identical output, so 'reliable' here is an empirical claim with a date on it. What we carry forward is the finding the instruments were built to catch: a system can be perfectly steady and still be deciding coin flips in silence. If an AI system inside your operation produces rankings, or anything else a steadiness check has blessed, and nobody has written the answer key, let's talk.