First-pass yield for AI systems

Ask an AI programme how it is going and you will hear volume. Requests served, documents processed, drafts generated, users onboarded. These are easy to count and they rise reliably, which is exactly why they get reported.

None of them tell you whether the work was any good. A system that generates a thousand drafts of which four hundred need substantial human rework is reported identically to one that generates a thousand drafts of which fifty do. The second system is worth several times the first and the dashboard cannot tell them apart.

Manufacturing settled this argument a long time ago, and the answer transfers cleanly. First-pass yield is the share of units that clear the process correctly the first time, without rework, rejection, or a second attempt. It is a quality measure rather than a throughput measure, and it is the number that determines whether automating something actually saved anything.

An AI pipeline has the shape of a production process. Work enters, passes through stages, and either clears or comes back. That is the structure first-pass yield was built for, and the mapping needs no forcing.

The unit is one piece of work: a draft, an extraction, a classification, a code change, a resolved ticket. It clears if it is accepted and used as produced. It fails first pass if a human had to materially fix it, if it was rejected, or if it had to be regenerated.

Two properties make it more useful than the alternatives. It is a ratio, so it does not improve merely because volume grew, which is the defect in almost every AI dashboard. And it is instantly legible to anyone who has run operations. You can put first-pass yield in front of a plant manager, a shared services lead, or a claims director and they will know what it means and what a bad number implies, which is not true of a token count or an eval score.

The reason to measure this rather than volume is that the rework is where the savings go to die.

A generated draft that a human substantially rewrites has not saved the time it appears to have saved. In some cases it costs more than starting fresh, because the reviewer has to read it, decide what is wrong, and then rewrite, which is three activities where there used to be one. Anyone who has edited a bad draft knows this in their body; the dashboard does not.

So the honest business case is not volume multiplied by time saved per unit. It is first-pass yield multiplied by time saved on a clean unit, minus the cost of handling everything that failed first pass. Programmes that report the first number and experience the second are the ones that produce impressive slides and no observable capacity.

This also reframes what improvement means. Raising first-pass yield from 60 to 80 percent halves the rework queue. Doubling throughput at 60 percent yield doubles it.

The measurement requires one thing most AI deployments lack: a defined accept event. You cannot compute yield if nobody records whether the output was used as produced.

The cheapest instrumentation is usually already available. If a human edits the output in a system you control, capture whether it was edited and by how much. If a reviewer approves or rejects, capture the disposition and the reason. If the output is a code change, the gate result is your accept event. If it goes to a queue, whether it came back is the signal.

Three cautions from doing this badly.

Define materially fixed before you start, and write it down. A typo correction is not a first-pass failure; a changed conclusion is. Without a definition your yield number will drift as reviewers interpret the threshold differently, and drift is indistinguishable from improvement.

Measure by segment, not in aggregate. A single blended yield across every use case hides the one that is failing, and the aggregate will look acceptable long after a specific workflow has become useless.

And distinguish a measured zero from an uninstrumented one. A workflow reporting no failures because nobody records dispositions looks identical on a chart to one that is genuinely clean. If you take nothing else from this: an instrument should be able to tell you it is not currently instrumented.

Low first-pass yield is a symptom with a small number of causes, and they need different responses.

The task is underspecified. The system produces something reasonable and not what was wanted, because what was wanted was never written down precisely enough to produce. This is the most common cause and it is not a model problem. It is the same defect as a work instruction that assumes the operator knows what good looks like.

The inputs are inconsistent. The process was never standardised before it was automated, so the system faces variation no rule can absorb. This is the oldest lesson in process engineering arriving in new clothing: automating an unstandardised process industrialises its variation.

The acceptance bar is undefined. Reviewers disagree about what passes, so the same output clears with one and fails with another. Your yield number is then measuring reviewer disposition rather than system quality.

Only after those three is it worth asking whether the model or the prompt is the problem. Teams reach for that explanation first because it is the one with a satisfying technical fix, and it is the least common cause of a bad number.

If low yield usually means the task was underspecified or the process was never standardised, then the work that raises it is not AI work. It is the work of defining what good output is, in enough detail that it could be produced deliberately.

That is unwelcome, because it is slower than tuning a prompt and it looks less like progress. It is also the reason a lot of AI pilots stall at a demo: the demo runs on the clean case, and the clean case is the only one anybody specified.

The order that works is the old one. Eliminate what should not exist, simplify what remains, standardise what is left, and only then automate. First-pass yield is what tells you whether you actually did the third step or skipped it, because a process that was never standardised produces a yield number that will not climb no matter what you do to the model.

If an AI programme is going to report a single figure to people who make funding decisions, first-pass yield is a better candidate than any of the usual ones.

It resists the failure mode of volume metrics, which improve when nothing improved. It is legible to operators without translation. It points at a specific, diagnosable set of causes when it is bad. And it is denominated in the thing the investment was supposed to produce, which is work that does not need doing twice.

Report it by segment, define the accept event before you start, and say plainly which workflows are not instrumented rather than leaving them out of the average.