The Governed Agent Stack

In the programmes we have worked on and the ones practitioners describe publicly, the binding problem is rarely model quality. It is that nobody defined who is allowed to see which data, that there is no gate a change has to pass, that nothing can prove to an examiner what the system saw, that the compliant path is slower than the workaround, or that cost grows with adoption until success and budget failure become the same event.

Every one of those is an architecture problem, and none of them is solved by choosing a better model. This is the reference architecture we use: nine controls arranged in three bands. Reach makes the platform usable. Control makes it safe. Proof makes it true.

The bands are a dependency chain rather than a menu. Reach without Control is a liability, because you have made an ungoverned capability easy to get to. Control without Proof is a belief, because you have written down what the system is supposed to do and never verified that it does it. Almost every programme we see has built the first band well, the second band partially, and the third band not at all.

Routing by intent. A user should not have to know which of forty capabilities to invoke. If discovery is the user's problem, adoption dies quietly and the platform gets blamed for being unhelpful when the real defect is that nobody could find the right door. Route by what the person is trying to do, not by the name of the tool.

Prevents: an internal platform nobody adopts. Costs: a routing layer that has to be maintained as capabilities change. You would know it broke if: usage concentrates on two or three capabilities while the rest go untouched, which usually means people found one door and stopped looking.

A capability registry. Every capability versioned, owned, and discoverable, with a record of what changed and when. This is unglamorous and it is what makes the rest auditable, because you cannot explain a past decision without knowing what the system could do on that date.

Prevents: an unanswerable question about what the system was capable of fourteen months ago. Costs: the discipline to register a capability at the moment you build it, which is the moment you least want to. You would know it broke if: someone asks when a capability changed and the answer requires archaeology.

Untrusted-input isolation. The moment an agent can both read external content and take action, whoever controls that content can potentially direct the agent. Email bodies, uploaded documents, scraped pages, transcripts, third-party API payloads. There is no reliable model-level fix for this, which is why the control has to be architectural.

The pattern has four parts and all four carry weight. The reader runs as a separate agent, so an injected instruction lands in a context with no authority and no history. It holds read-only tools, so even a fully successful injection reaches something that cannot send, write, or purchase. It returns structured output, so its response is parsed as data rather than executed as direction. And it is told explicitly that everything it reads is data and never instructions.

The point worth internalising: you are not trying to stop the injection from succeeding. You are arranging for its success to be irrelevant.

Prevents: an attacker steering an agent through content your organisation did not write. Costs: an extra hop, and the discipline of briefing the isolated reader rather than letting the main agent read directly. You would know it broke if: an agent that reads external content is holding a write-capable tool. That is a one-line audit and it should be run on a schedule, not on a hunch.

A provenance gate. Nothing external-facing ships with a claim that does not trace to a source line or an explicit human confirmation. No creative-interpretation tier. No implied-by-other-facts tier. Unsourced claims are dropped rather than softened, and the deliverable carries a map pairing each claim to its source. In financial services this is model risk management applied to generated text, and it is the control that decides whether the technology can touch anything client-facing at all.

Prevents: a fabricated number reaching a client, a filing, or a regulator. Costs: real throughput. Some outputs get smaller. You would know it broke if: a deliverable ships whose provenance map has fewer entries than it has claims.

Tiering with a quality floor. Route cheap work to cheap models. That part is obvious and every vendor will sell it to you. The part that matters is the floor: security review, adversarial testing, architecture decisions, and final synthesis may never be downgraded, regardless of budget pressure. Cost optimisation that can silently degrade a security review is not optimisation. It is undisclosed risk transfer.

Prevents: a budget decision quietly becoming a safety decision. Costs: your cheapest possible bill is not available to you. You would know it broke if: a cost-reduction initiative cannot name which categories it is forbidden from touching.

Promotion gates. A change passes defined checks in a defined order before it reaches production, ordered by cost of discovery so a cheap gate rejects what an expensive gate would waste effort on. A failure returns to the author rather than proceeding. The single test of whether you have a gate: a gate that can be waived by the person it blocks is not a gate, it is a suggestion.

Prevents: an unreviewed change reaching users. Costs: latency on every change, which is exactly the cost people try to negotiate away. You would know it broke if: the gate is available but optional. Availability is not enforcement.

Autonomy questions usually get argued in terms of capability, as though the issue were whether the agent is good enough. That is the wrong axis. The question is what happens when it is wrong, which means the axis is reversibility.

Reversible and contained actions should run autonomously. Log them, do not gate them, because a gate on a reversible action is pure friction and friction is what sends people around the paved road. Anything irreversible or wide gets propose, confirm, execute, with a dry run that shows exactly what would happen and a human in the path before it does.

This is also the answer to the most common objection, that governance slows everything down. It does not, if you apply it by blast radius. Most agent actions are reads, drafts, and analysis, and those should be untouched. The gate belongs on the small number of actions that send money, delete records, or reach a customer.

Prevents: an autonomous action nobody can undo. Costs: a confirmation step on the actions that deserve one. You would know it broke if: an irreversible action path exists with no dry run, which is worth enumerating deliberately rather than discovering.

An evidence rule. No completion claim without fresh proof produced in the current pass. A prior run's passing test is stale. A component reporting its own success is a claim, not evidence, and the diff is the evidence. We learned this one the hard way: agents in our own pipeline reported clean work that was not clean, and the fix was structural rather than corrective. The check moved from the agent that produced the work to the orchestrator running against the assembled output.

Prevents: a system that reports success it cannot demonstrate. Costs: verification work that feels redundant right up until it is not. You would know it broke if: your completion signal is something a component said about itself.

An evaluation harness. A fixed set of realistic tasks, written rubrics, a held-out split so prompts cannot be tuned to the test, and an independent judge. Without it, every prompt change ships on the author's confidence, which is grading your own homework. This is the promotion gate for AI-specific changes and it is the single most commonly missing control in the field.

Prevents: an unmeasured change to a non-deterministic system. Costs: task authoring, which is genuine work, and the runtime of the judge. You would know it broke if: the harness exists and has not been run. Ours went six weeks, and when we did run it, the number it produced was measuring a defect in the harness rather than a defect in the agents.

Telemetry and a ledger. What ran, what it cost, what changed, what was retrieved, what the human decided. The evidence has to be a byproduct of running the system rather than something assembled afterward for an examiner, because a trail reconstructed on request is not a control.

Prevents: an unanswerable audit. Costs: storage and the discipline to instrument before you scale rather than after. You would know it broke if: you cannot distinguish a measured zero from an uninstrumented one.

The pattern in every entry above is deliberate. A control is not a thing you install. It is a thing that can quietly stop working, and the failure mode is silence rather than error.

So the question we would now put to any AI control anyone shows us, including our own: when did this last produce a result you independently confirmed was correct, and how would you know if it had quietly stopped working?

If the answer to the first half is a date, the control is real. If the answer is a shrug, or a reference to the fact that it is configured, you have a diagram rather than a control. The distinction matters most exactly where it is hardest to see, because a control that has silently failed looks identical to one that has never been needed.

Do not build this in order. Build it in risk order, which for most organisations means the boundary first.

Start with untrusted-input isolation if any agent touches content your organisation did not write, because that is the control whose absence is exploitable rather than merely untidy. Add the provenance gate next if anything the system produces reaches a client or a regulator. Then promotion gates, then the evidence rule, then evaluation.

Reach can wait, which is counterintuitive because it is the band that demos well. A capability nobody can find is a wasted investment. A capability everybody can find and nobody governs is an incident.

And build Proof earlier than feels necessary. It is the band with no immediate symptom, which is precisely why it gets deferred, and deferring it means every improvement you make for the next year is unverified. The cost of adding it later is not just the build. It is that you cannot tell which of your past changes helped.

On effort, since every control above describes its cost qualitatively: the boundary and the provenance gate are each a matter of days for a team that already knows its data flows, because both are mostly a matter of deciding what an agent may hold rather than writing much. Promotion gates take longer, not because the logic is hard but because agreeing what blocks a release is a political exercise. The evaluation harness is the one genuine build, and the expensive part is authoring realistic tasks rather than writing the runner. Telemetry is cheap if you do it before you scale and painful afterwards.

If you want a concrete first step: take one agent already running in your organisation and answer the three questions for it. What failure is each of its controls preventing, what does that control cost you, and when did it last produce a result someone independently confirmed. Most teams cannot answer the third question for a single control, and that answer is the start of the work.