The blindfolded judge: what our eval harness was actually measuring

The same input can produce a different output twice in a row. That single fact breaks the way software teams normally know whether a change was an improvement. You cannot diff the behaviour of a prompt the way you diff a function, and you cannot run it once and conclude anything.

So you build an evaluation harness. A fixed set of realistic tasks, a written rubric for each, and an independent judge that scores the output against the rubric. Run it before and after a change and the delta tells you whether you helped or hurt. Without it, every prompt edit ships on the author's confidence, which is the definition of grading your own homework.

We built one for our internal agent system. Task sets, written pass and fail criteria, a held-out split so the prompts could not be tuned to the test, and a judge running in a fresh process per task so no task could contaminate the next. On paper it was the right design. We are reasonably proud of the design. That turns out to be part of the problem.

Precision matters here, because this is an article about measurement. The harness was not idle for six weeks in the sense of never executing. It ran twice in plumbing mode, once in late June and once in mid July, using a deterministic self-test grader that exists only to prove the runner works. What it had never done, in the six weeks between being wired to a real judge and the day we describe here, was grade anything for real.

When we finally ran it against a real judge, one agent family scored 0.227 out of 1.0. Every task failed. The judge's written feedback was specific and damning: the agent did not route the work correctly, did not read the files it was told to read, formed generic theories instead of concrete ones, and asked clarifying questions instead of doing the job.

That is a coherent story. It reads like an agent with a genuine orchestration problem. A team looking at that dashboard would open the agent's prompt and start fixing it.

Before changing the agent, we read the judge's complaints more carefully. Every failure shared a shape: the agent said it could not find the files. It kept asking where the repository was.

The harness builds a prompt for each task from three pieces: the task instruction, the surrounding context, and the files the task supplies. We checked the function that assembles it. It used the instruction. It used the context. It never included the files.

The file array was silently dropped. Every candidate the harness had ever produced was answering blindfolded: asked to review a diff it was never shown, asked to debug source it was never given. The agents had been declining to invent code they could not see, which is exactly the behaviour we want, and the rubric had been scoring that correct behaviour as failure.

Every score the harness had ever recorded was not low. It was meaningless.

We added the files to the prompt, with an explicit instruction that the supplied files are the repository for that task. We populated fixtures on a task whose rubric demanded the agent read two source files that had never been attached. We added regression tests, because the reason a dropped field survived this long is that the test suite asserted the instruction and the context and had never asserted the files.

Then we ran it again. The code-review family went from 0.200 to 0.940. Every task passed. The judge now describes an agent that catches a planted authorization bypass with exact file and line numbers, names the concrete exploit path, and correctly declines to manufacture nitpicks on the clean diff it was also shown.

Nothing about that agent changed. Not one word of its prompt. It had been performing well the entire time, and our instrument had been reporting it as broken.

One disclosure, because a single flattering number is how this kind of article usually cheats. The other family we ran did not resolve as cleanly. It moved from 0.200 to 0.460, one of its tasks timed out during production and was reported as pending rather than passed, and two of its failures survived the fix and are real findings about that agent's behaviour. The instrument defect was common to both families. Only one of them turned out to be fine underneath it.

This is the part worth carrying into your own organisation.

If the harness had crashed, someone would have fixed the harness. If it had returned nothing, nobody would have drawn a conclusion. Instead it returned 0.2 with articulate, specific, entirely wrong reasoning attached. That is far more dangerous, because it is actionable. It invites you to act.

Follow the action it invited. A team trusting that number would have edited the agent to stop asking for missing files and to proceed with a diagnosis anyway. They would have trained out the one behaviour standing between them and a confidently invented root cause. The metric would have improved. The agent would have become materially worse, and the dashboard would have called it progress.

A measurement system that is wrong in a plausible direction does not merely fail to help. It actively steers.

We have a rule for this, and we failed to apply it to ourselves. A dashboard showing zero failures means nothing until you know whether the system measured zero or measured nothing. The two look identical on a chart and mean opposite things.

Our eval score was in the second category and presented as the first. There was no field in the output that distinguished them, which is the real design flaw underneath the dropped variable. An instrument should be able to report that it is not currently instrumented.

The practical version, for any AI system you are asked to trust: before accepting a quality metric, ask what would have to be true for that number to be produced by a broken measurement rather than a broken system. If you cannot answer, the number is decoration.

The harness existed and did not run. There was a reminder in place that fired whenever an agent prompt was edited, deliberately non-blocking so it would not interrupt routine work. It fired. Nobody ran the eval.

A control that is available but optional is not a control, it is a suggestion. This is the most ordinary failure in AI governance and it has nothing to do with AI. The policy exists, the tool exists, and the gate is never actually in the path of the work. When the review does eventually happen, it happens because someone decided to look, which means it happens least often when things are busiest.

We are not going to claim we solved this. We ran the eval, pinned a baseline so future runs have something to compare against, and the wiring that would make it a blocking gate is still open. Naming it is worth more than pretending otherwise.

One more observation from the same afternoon, and one correction we owe you about it.

We re-ran the judge on identical candidate answers, changing nothing at all. The family score moved from 0.940 to 0.880. Our first instinct was to call that a noise floor of roughly 0.06 and tell you to discount any smaller delta.

That would have been the same error this entire article is about. Two runs is not a variance estimate. A single pair of numbers tells you that variance exists and tells you almost nothing about its size, and presenting one spread as a floor would be over-reading a measurement in precisely the way we just spent two thousand words warning against. Writing it down is what caught it.

So the honest version. We observed a 0.06 spread on unchanged inputs, which is enough to establish that this judge is not deterministic and not enough to quantify how much it moves. Establishing that requires repeated runs and a distribution rather than a pair. Until we have run it, we do not have a threshold, and neither does anyone else who has sampled twice.

The transferable rule is weaker than the one we wanted to write, and it is the one that is actually true: if you use a model as a judge, measure its behaviour on unchanged inputs enough times to know the shape of its variance before you interpret any single delta. An eval you over-read is a slower path to the same false confidence.

The uncomfortable summary is that we built the thing designed to catch silent failure, it failed silently, and we found it the only way anyone ever finds this class of defect, which is by actually running the check instead of trusting that it was configured.

It is worth saying plainly what that does and does not mean for anyone weighing whether to take our advice. A firm that has never found a defect in its own controls has either not looked or is not telling you. The question worth asking of any advisor is not whether their instruments have failed. It is whether they run the checks that would reveal it, and what they do on the day one comes back wrong. This is what we did on that day, written down while it was still embarrassing.

We think that is the useful version of this story rather than an embarrassing one. Governance programmes are usually presented as a set of controls placed around a risky system. What this cost us is the reminder that the controls are themselves systems, with their own defects, their own silent modes, and their own need for someone to ask when they were last verified end to end.

The question we would now ask of any AI control anyone shows us, including our own: when did this last produce a result you independently confirmed was correct, and how would you know if it had quietly stopped working?