Where LLMs actually fail: composition, not retrieval

In our last article we argued that the human gate belongs where a wrong-but-confident output would cause harm before anyone catches it. That rule tells you which workflow gets a gate. It does not tell you where inside the workflow the gate earns its keep. A workflow is not one task. It is a mix of lookups, judgments, and drafting, and a gate placed on the wrong task type buys review volume without buying safety.

This article moves the question one level down, from which workflow to which task type inside it. It rests on a small piece of direct evidence: a controlled stress test our founder ran during his graduate study at Purdue, on the kind of document our own rental operation produces every week. The result was lopsided. On the same document, with the same model, cued retrieval was near-perfect, and every verified error appeared during open composition. The model did not misremember anything. It added things.

The test document was a synthetic operations log, written for the test, and we want to be explicit about that word. No real guests or bookings, and nothing pulled from production systems. It was roughly 1,800 words of weekend activity for a short-term-rental operation: check-ins and checkouts, maintenance incidents, supply orders, fee decisions. The shape of a document our own agents read every day, with none of the content.

It was also built to be maximally confusable. Unit numbers R-114, R-141, and R-411 recur through the log. A door code changes from 4471 to 7414 and then reverts to 4471, so the right answer depends on reading the whole timeline, not the first or the latest mention. A guest credit and a separate fee carry amounts that are digit-transposes of each other. Two purchase orders, P-2201 and P-2210, differ by one swapped digit. If a model were going to confuse similar facts, this log was designed to make it happen.

The protocol was three runs, each in a fresh session with no memory of the others, on the free consumer tier of Claude, the same model family our own operation runs on. Run one asked pointed questions of a shorter early version of the log; the model caught every trap, so the log grew to full length. Run two asked pointed questions of the full log. Run three handed the model open composition work: write the Sunday morning shift brief, and draft the guest messages.

The pointed questions were the part built to be dangerous, because that is where the confusable details live. They were not dangerous at all. The model walked the door-code timeline correctly, 4471 to 7414 and back to 4471, without anchoring on either end of it, and it kept the credit and the fee straight even though the amounts were designed to be mistaken for each other. It flagged the transposition risk between the two amounts on its own. Nothing in either question run crossed the unit numbers or the purchase orders. Across both question runs, retrieval never failed.

One caution before anyone reads that as a blanket endorsement of Q&A. The research that predicts this result also names the risk this short test did not trigger. Liu and colleagues showed that models use long contexts unevenly, and that facts buried mid-context in long stitched documents are the likeliest silent retrieval misses. A compact log with pointed questions plays to the model's strength. Retrieval earned its clean score here; it has known failure conditions of its own at longer range.

Run three produced three verified errors, and their shape is the finding. None was a retrieval miss. The unit numbers, the credit and the fee, the door codes: the confusable material came through clean again. All three errors were additions, statements the log never made, delivered in the same confident prose as everything else.

First, supplies the log said were ordered became 'on hand' in the drafted brief. The same response then advised confirming delivery, contradicting itself within its own text. Second, the model invented a gender for Riley, a synthetic guest the log never described. That slip reproduced in two independent fresh sessions, which makes it hard to dismiss as a one-off. Third, where the log was silent on whether to charge a fee, the drafted brief turned that silence into a directive not to charge one, while a fee precedent sat one sentence earlier in the same note.

So the score reads: pointed questions, zero retrieval errors; open drafting, three errors, all unsupported additions. The task type, not the difficulty of the material, decided where the errors landed. The confusable details were the hard part of the document, and the model handled every one of them. What it could not handle was silence. The moment the task required deciding what belongs in the output, it filled the gaps with plausible material the source never supplied.

Three papers, read alongside the test, explain why the split looks this way rather than some other way.

Liu and colleagues, in 'Lost in the Middle,' show that a model's attention retrieves what a cue points at. A pointed question supplies that cue and anchors the model to the source line. An open drafting task supplies no cue at all, so nothing anchors the model's choices about what belongs in the output.

Bender and Koller, in 'Climbing towards NLU,' argue that fluent output is decoupled from grounded meaning. An unsupported addition is exactly that: form without grounding. 'On hand' is a plausible continuation of a supply note, not a claim checked against anything.

Shanahan, in 'Talking About Large Language Models,' completes the picture. The model composes the statistically likely brief, not the verified one. Saying it 'knows' the supplies arrived is a category mistake; it produced the sentence a brief like that usually contains. Where the source is silent, on a fee decision or a guest's pronouns, composition fills the silence with the distribution's most likely value.

Together the three name the danger precisely: the risky outputs are the fluent ones. The model writes well about things it never grounded. A reviewer skimming the drafted brief finds nothing that reads wrong, because nothing reads wrong. It is only wrong against the source.

Here is what this does to the gate-placement rule. The original form: put the human gate where a wrong-but-confident output would cause harm before anyone catches it. The refinement: inside any workflow, that harm concentrates in the composition tasks. Cued lookups, the Q&A the same model handled near-perfectly, do not need the same gate. Drafting, synthesis, and summarization do, because that is where the wrong-but-confident additions appear.

The test left us with a three-part spec. Composition gets claim-level citations: every operational claim the model composes, a code, a time, an amount, a pronoun, must trace to a source line. The human check then becomes a scan, not a re-read: instead of re-reading the whole source against the draft, the reviewer walks the citation trail. And evaluation has to hit the composing task directly, because Q&A performance overstates the safety of drafting. This failure class only appears when the model must decide what belongs in the output.

That last point is the one we would put on a wall. If you evaluate an AI writing system by asking it questions, you are testing the task it is good at and shipping the task it is bad at.

This is not abstract for us. Our own rental operation runs day to day on Claude-based agents inside Slack, a system we call RebornOS. Today those agents triage guest messages and incidents, handling the routine cases and escalating the ones that need a human decision, and they assemble month-end reporting from the ledger as it grows. The natural next step for a system like that is the one the test exercised: letting agents compose the guest replies, the incident summaries, the shift handoffs. That is exactly the workload where unsupported additions appear, and it is why our gates sit where they do. Agents work the volume; people review before anything consequential ships.

The test sharpens what those gates have to check before any agent is handed the drafting itself. The claims that matter in a guest message, a door code, a check-in time, a fee decision, are exactly the claim types the test showed the model will fill in when the source is silent. We already log what each agent read, what it decided, and why. The finding turns that trail into a specification: every operational claim in a composed message should trace back to a source line, so the human review is a scan of the trail rather than a re-read of the whole thread. That is the standard we will hold any composing surface to, and the test is the reason.

Now the honest part. This was three runs, on one synthetic log, on one consumer tier of one model family, each in a fresh session. The Riley slip reproduced across two independent sessions, which gives it weight. The class-level claim, that errors concentrate in composition rather than retrieval, fits the published research and is consistent with how we operate our own agents. It still needs replication across models and documents before anyone should treat it as doctrine. We treat it as a working hypothesis strong enough to design around, because the cost of designing around it is low and the cost of the failure it names is not.

There is also an open question we have not answered. Does forcing claim-level citations suppress the additions, or relocate them? The spec assumes that requiring a source line for every claim prevents unsupported additions. A model might instead cite lines that do not support the claim. That is the next test to run, and we will publish what we find.

In the meantime the operational rule stands on its own: gate the drafting, not the lookup, and make composed claims show their sources. If an AI system writes on your behalf, test the writing, not the Q&A. If you want that test run against your operation, let's talk.