What the format ate: when a better prompt starts deleting things

Three articles in, the rule has a shape. Put the human gate where a wrong-but-confident output would cause harm before anyone catches it. Inside a workflow, that harm concentrates in composition rather than in cued lookups. And a consistency check cannot tell you whether the composed answer was right, only that the machine was steady while producing it. The next move is the one every practitioner reaches for on their own: if composition is where the model slips, constrain the composition. Give it a role, a format, a policy for how to rank, and a rule that every claim has to cite its source.

So we did exactly that, on the workload our own rental operation runs constantly, during our founder's graduate AI coursework at Purdue. The constrained prompt worked. It also introduced a failure the loose prompt never had, and it is the kind nobody catches by reading the output, because the output looks finished. That is what this article is about.

The task was small and it is real work: take a batch of de-identified guest reviews for a single rental property and reduce them to about three recurring themes, each with one action attached. Ten reviews, synthetic, written for the test. No real guest data, no real property, and we want to be explicit about that. The same ten reviews fed both runs, so the prompt was the only thing that changed between them. The model was Claude, pinned by version, each run in a fresh session, the same protocol discipline as the last test.

Two prompts. The first was the lazy one, the one anybody actually types on a Tuesday: 'Summarize these guest reviews and tell me what to fix.' The second was engineered. A role, background on the property and the operator's constraints, a clear task statement, a fixed output format, one worked example from a different property, and then a round of iteration on what came back.

Before we argue from the result, the honest version of it, because the tidy story is not what happened. The plain prompt got the hardest call right. A loose front door lock appeared in exactly one review out of ten. A three-mention inconvenience appeared, as you would guess, three times. The plain prompt put the lock at the top of its list, under its own heading: 'fix first despite one mention.' Safety over frequency, with nobody telling it to. This is not a story about a bad prompt getting rescued.

It failed in three other places, and the first one is the most interesting. It built a table that ranked by frequency, scored the lock '1 (but severe)', and then wrote a prioritized list underneath that overrode the table it had just built. It reached the right answer by contradicting its own structure. That is a good outcome produced by an unreliable mechanism, and an unreliable mechanism is precisely the thing you cannot schedule for next week.

Second, it stated an inference as a finding. Three reviews mentioned access problems, and the output announced that 'This points to a systemic sync issue between your lockbox/smart lock and whatever app or messaging tool generates codes.' Nothing in the reviews says that. Nothing in the reviews describes the property's access setup at all. The model invented an architecture and handed it back as a diagnosis, in the same confident register as everything it had actually read. Readers of the composition article will recognize the shape: the slip is an addition, never a retrieval miss.

Third, the actions were advisory rather than fundable. 'Worth a maintenance check before peak summer bookings' is a sentence with no owner and no date in it. Nobody can put that on a board and nobody can close it.

The engineered prompt supplied all the usual parts, and one part did most of the work. A severity ladder, written down as policy: safety and habitability outrank a guest being unable to complete a core action, which outranks comfort, and severity outranks frequency in every case.

That ladder is not recoverable from the reviews. No amount of clever wording gets a model there, because it is not a fact sitting in the text waiting to be found. It is an operator's decision about what the business will not tolerate, and it exists only if somebody writes it down. Once it was written down, the lock ranked first structurally instead of by argument. Same answer as the lazy prompt produced, different mechanism, and the mechanism is the part you can rely on.

The second rule generalizes further than the first. Every claim had to cite the reviews it rested on. Anything establishing severity had to quote the exact words that established it. And anything the model concluded but the reviews did not literally state had to move out of the findings into a separate Inferences block, along with a note on what would confirm it. Do not present an inference as an observation. That one instruction is the difference between a finding and a guess wearing a finding's clothes.

The engineered prompt capped the output at exactly three themes. Three is a good number for an operator. It fits on a board, it fits inside a morning meeting, and it forces a choice instead of a list. It also silently deleted a real recurring theme.

Air conditioning was raised by two separate guests, with summer coming. It fell off the bottom of the table and nothing in the output said so. No note, no 'also mentioned', no count of what did not make the cut. The table looked complete, because a table with three filled rows always looks complete. That is the finding, and it deserves to be stated flatly: a capped output looks finished precisely when it is not. The constraint that made the output usable is the same constraint that made it lossy.

The fix is small, and we now treat it as mandatory. Add a named overflow slot to every capped output, a Watchlist, holding anything with two or more mentions that did not make the ranking. When we added it, the air conditioning theme came back with a re-score trigger attached, a stated condition under which it should be promoted. The cap stayed. The loss did not.

The iteration round caught a second thing, and it is the more uncomfortable of the two. The model had written 'INFERENCES: None' at the bottom of an output whose recommended action told us to sync three code systems, as if a shared root cause had been established. The fabricated diagnosis had not disappeared when we added the evidence rule. It had moved out of the findings, where the rule was watching, and into the action, where it was not.

When we put that to the model directly, it found a sharper version than the one we brought. Grouping the three access failures into a single theme is itself the inference. One review described a code that did not match, one described contradictory instructions, and one described a code that stopped working outright. Three failures of access, three different mechanisms, and one theme heading quietly asserting they share a cause. An evidence rule that polices sentences will not catch an inference that lives in a heading.

There is a widely reported finding that prompts matter as much as models, and the study behind it is more specific than the headline. It is Jahani, Manning, Zhang, TuYe, Alsobay, Nicolaides, Suri and Holtz, 'Prompt Adaptation as a Dynamic Complement in Generative AI Systems,' 2026. Two preregistered tasks, 3,750 participants, nearly 37,000 prompts.

The split runs along one axis. On the task with fixed evaluation criteria and an unambiguous goal, prompt adaptation accounted for roughly half the performance gain that came from upgrading the model outright. On the open-ended creative task, where the space of acceptable outputs is effectively unbounded and quality is subjective, the gains came primarily from model capability and prompt adaptation played a limited role. The authors also report that automated prompt rewriting cannot generally substitute for human adaptation, which is worth sitting with if the plan was to have a model write your prompts for you.

So the question to ask before investing in prompt engineering is not how much prompts matter in general. It is whether this particular task has a checkable right answer. If it does, prompt work is among the cheapest gains available to an operation. If it does not, buy the better model and keep the human.

We ran the Claude-based agents inside our own rental operation, the Slack system we call RebornOS, through that axis. The sort was clean enough to change how we build.

Review theming and message triage are bounded. Fixed categories, a severity policy we wrote ourselves, and an answer that can be checked against the source. This is where every method in this article pays for itself, and where the prompt deserves real investment: the severity ladder, the citation rule, the inference block, and an overflow slot on every capped output without exception.

The morning operations brief is bounded too, and it is the highest-risk item on our list, because a brief is a capped output by definition. Somebody decided how long it should be before anyone knew what it would have to say. A brief that drops the one thing that mattered reads exactly like a brief that had nothing to report, and nobody in the meeting can tell those two apart. That workload needs the Watchlist pattern more than anything else we run.

Guest reply drafting is unbounded. Tone, judgment, reading a particular guest's mood on a particular day, no single right answer to check against. We expect a much smaller return on prompt engineering there, and the discipline that matters is not letting a polished prompt stand in for the human gate. That is the first article's rule still doing the work, and prompt craft does not replace it.

Now the honest part, which is becoming a habit in this series. Two runs of the identical engineered prompt disagreed with each other. One listed its inferences, the other said 'None'. So the format constraint tightened the shape of the answer and not its substance, and that is a smaller claim than 'the engineered prompt is better'. It is the claim the evidence actually supports.

We lost things too. The plain prompt flagged what was already working well and warned us not to break it while fixing everything else. Our format has no slot for that, so it vanished. On that one point, loose prose was sharper than our table, and we have not fixed it yet. There is also the arithmetic: roughly 600 words of prompt to produce 250 words of output is a bad trade for anything you run once. The engineering pays when the prompt becomes an asset that runs every week, feeds an agent, and gets versioned like code. For a question you ask a single time, just ask the question.

The three findings stack, and the stack is the point. Composition is where models add things the source never said. Consistency does not prove correctness. And the constraints you add to fix both of those will delete things quietly unless you build a place for what they delete. If an AI system in your operation produces a capped list, a top three, a summary, a brief, and nobody has asked what fell below the line, let's talk.