Building an Agent That Can't Be Hijacked: The Design Rules

Table of Contents

TL;DR

• I spent a previous post arguing that prompt injection detectors don’t work and can’t be made to work by training harder. That post ended by naming the alternative — CaMeL, Fides, the design-pattern literature — and then stopping. This is the follow-through.
• There are at least eight frameworks telling you how to secure an agentic system. The most common mistake is treating them as competitors. They sit at different layers and answer different questions: a risk taxonomy is not a control catalogue, a management system is not an architecture, and a regulation is neither.
• Strip the vocabulary away and they converge on five things: least privilege, provenance and isolation of untrusted content, human oversight at consequential decisions, auditability, and supply chain.
• They also genuinely contradict each other in at least four places, and those places are where real designs go wrong. The most expensive one: the law tiers risk by application domain, while agent security is determined by capability and blast radius. Those two axes are close to orthogonal.
• The defense literature — Dual LLM, CaMeL, Fides, Progent — and the eight frameworks turn out to cover disjoint parts of that list. The papers mechanise provenance, capability and auditability and impose no obligations; the standards impose obligations for human oversight and supply chain and mechanise nothing.
• From the convergence I derive four properties that a triage agent will enforce deterministically, outside the model. Part 2 builds the interpreter that enforces them, attacks it, and publishes the numbers — including how much utility the defense costs.

The Problem, Restated in One Paragraph

The argument of the detector post was that filtering is the wrong shape of solution: there is an impossibility result under standard cryptographic assumptions, there is a measured security-fidelity tradeoff, and there is a decade of evidence that defenses evaluated against fixed attacks collapse against adaptive ones. I’m not relitigating it here. The relevant conclusion is narrow and it’s this: if your security property depends on a model classifying its input correctly, you don’t have a security property. You have a probability, and an adversary who gets to move second.

So the question this post starts from is the constructive one. If not a classifier, then what — and does anyone actually agree on the answer?

What the Standards Actually Say

Eight documents get cited in this space. Here they are with the only two questions that matter: what does it answer, and what does it refuse to give you.

The table is the point of this section. Read down the last column and the picture is obvious: no single document in this list is sufficient, and none of them is trying to be. A team that adopts OWASP’s agentic top ten has a list of things that can go wrong and no controls. A team that certifies against ISO 42001 has a defensible process and no architecture. A team that maps to SP 800-53 has controls designed for principals whose identity is stable — which an agent acting under delegated authority is not.

The failure mode I keep seeing is a team picking one, treating it as complete, and shipping.

OWASP’s agentic list as the backbone

I’ll use the agentic top ten as the spine for the rest of the post, because it’s the only one on the list built specifically around the things that make an agent an agent: it holds memory, it plans, it calls tools, and it acts with authority someone delegated to it.

Its ten categories cover planning, tool use, identity, supply chain, code execution, memory, inter-agent communication, cascading failures, human-agent trust, and rogue agents.

Two observations before moving on. First, six of those ten are not model problems at all — they’re systems problems that happen to have a model in the loop. Identity, supply chain, code execution and memory are things we’ve known how to reason about for decades; what’s new is that the thing holding the privilege now takes instructions from text it read on the internet. Second, the list gives you no ordering. Every entry looks equally urgent on paper, which in practice means teams do the cheap ones.

What They All Converge On

Take the vocabulary away and the eight documents agree on five things. This is the useful output of reading all of them, and it’s short enough to hold in your head:

1. Least privilege and capability limits. The agent should be able to do the narrowest possible set of things, and the bound should not depend on the agent’s own judgment about what it should do.
2. Provenance and isolation of untrusted content. Data the adversary can influence must be identifiable as such, and it must be structurally separated from instructions.
3. Human oversight at consequential decisions. Someone accountable approves the actions that are expensive to undo.
4. Auditability. You can reconstruct what the agent did and why, after the fact.
5. Supply chain. Models, tools, prompts and connectors are dependencies with the same trust problems as any other dependency.

There is a detail in that list worth pulling out, because it’s the one place where the compliance world and the security world get the same thing for the price of one. Points 2 and 4 are the same mechanism. If you track provenance well enough to enforce a data-flow policy, the structure you built to do it is the audit record — not a log you write alongside the real work, but the actual thing that decided whether the action was allowed. Auditors want a defensible account of why an action happened. Provenance tracking produces exactly that, as a byproduct of enforcement. I’ll come back to this in Part 2, because it turns out to be the cheapest thing in the whole design.

Where They Contradict Each Other

This is the section that doesn’t get written, and it’s where designs actually go wrong. Four real conflicts:

1 · Human oversight is a control in one framework and a vulnerability in another. The regulatory instruments treat a human in the loop as a mitigation to be demonstrated. The agentic risk taxonomy treats human-agent trust as a risk category — because a human who approves twenty prompts a day stops reading them around the fourth. Satisfying the first requirement in the laziest possible way makes the second risk worse. A rubber stamp is compliance evidence and a security hole simultaneously, and no document tells you that because no document covers both layers.

2 · Certification measures consistency; adversarial security measures resistance. A management system rewards a documented, repeatable process. Security research rewards the assumption that your process is wrong and someone is optimizing against it. These are not opposed in principle, but they compete for the same budget and the same calendar, and one of them produces a certificate.

3 · The law tiers risk by application domain; agents are dangerous by capability. Regulation asks what the system is for — credit scoring, biometrics, employment. Agent security is determined by what the system can reach. A chatbot that falls outside every high-risk category but holds a shell and a cloud credential has a blast radius that a formally high-risk classifier emitting a score does not. These two axes are close to orthogonal, and only one of them is legally binding. Optimizing for the legal axis alone produces systems that are compliant and trivially dangerous.

4 · Control catalogues assume attributable principals. The access control, identification and audit families were written for a world where an action traces to an identity, and the identity is a person or a service with stable authority. An agent acting under delegated authority breaks this: the action was taken by the agent, authorized by the user, and caused by a sentence in an email that a third party wrote. Existing catalogues have no vocabulary for the third one, and it’s the one that matters for injection.

The practical consequence of all four: you cannot derive an architecture by compliance. You can derive one from the convergence, and then check it against the obligations. That ordering matters and it’s the opposite of how most organizations do it.

The Literature the Standards Don’t Cite

The section above ended on “you cannot derive an architecture by compliance.” That’s true and it’s incomplete, because there is a body of work that hands you an architecture — and not one of the eight documents cites any of it.

Five steps, briefly, since I covered them at length in the detector post:

• Dual LLM (Willison, April 2023) is the ancestor: a privileged LLM that calls tools but never sees untrusted content, and a quarantined LLM that reads untrusted content but cannot act.
• CaMeL (arXiv:2503.18813) makes it a mechanism — extract the control and data flow from the trusted query so untrusted data can never affect program flow, and attach capabilities so policy is checked at tool call time. It also states its price: 77% of AgentDojo tasks with provable security against 84% undefended.
• Fides (arXiv:2505.23643) does information flow control directly, with confidentiality and integrity labels and explicit declassification.
• Progent (arXiv:2504.11703) does privilege control: a symbolic policy over tool names and arguments, evaluated on every call.
• Design patterns (arXiv:2506.08837) names six topologies that trade agent generality for resistance, of which Plan-Then-Execute is the one this series ends up using.

Now put that list next to the convergence list from two sections ago, which is where it gets interesting:

Read the middle column downwards and the gap is loud. Every one of these mechanises points 1, 2 and 4. Not one of them touches point 3 or point 5. There is no paper here about how much human approval an architecture generates before the human stops reading, and none about the supply chain of the tools and prompts the agent depends on.

Which resolves the “no document is sufficient” complaint from earlier in a way I didn’t expect when I started reading. The two literatures are not rivals and they’re not redundant — they cover disjoint parts of the same list. The papers give you mechanisms for provenance, capability and auditability, and no obligations. The standards give you obligations for oversight and supply chain, and no mechanism for anything at all.

So the derivation in the next section takes both. P1 and P2 are CaMeL and Fides restated for one domain. P3 and P4 have no paper behind them — they exist because the convergence says an accountable human approves consequential actions and that capability must be bounded, and because nobody in the defense literature has published what those two cost.

That gap is why Part 2 measures them. It is also, as it turns out, where one of my four policies falls apart.

From Principles to a Design

Here’s the derivation. The five convergence points, applied to an agent that reads attacker-influenced data, produce four properties I can state precisely enough to enforce in code.

The agent for Part 2 is a SOC alert triage assistant. It reads an alert, pulls the raw log, extracts indicators, enriches them against third-party services, and produces a report. It is deliberately read-only: it does not block IPs, isolate hosts or disable accounts.

That last decision deserves a defense, because “read-only” sounds like I dodged the hard part. I didn’t. A read-only agent still performs an irreversible act every time it enriches an indicator. You cannot un-submit a hash to a public scanning service, and doing it tips off the adversary that they’ve been seen — a well-known operational problem that predates any of this. And a lookup against a domain the attacker controls is an exfiltration channel: the internal hostname goes out encoded in the subdomain. So the interesting security properties survive intact, in a setting where a mistake during my own testing can’t take down a network.

P1 is the one people get wrong, so let me be explicit about what it means, because it is stronger than it looks. It is not “don’t pass dirty data to tools”. It is don’t branch on dirty data. A line like:

if verdict == "malicious":
    escalate()

is a violation when verdict came from a model that read the attacker’s text — because the attacker chose the branch. Almost every hand-rolled “secure agent” I’ve read has some version of that line in it, and its authors would tell you they’ve isolated untrusted content. They’ve isolated it from the arguments and handed it the control flow.

The architecture

Two design decisions in that diagram are worth naming, because they’re what make the guarantee statable in one sentence.

The plan is validated in full before the first tool call. Not step by step with replanning. The entire program is parsed, checked against all four policies and approved before a single untrusted byte enters the process. This is a real restriction — a human analyst triages iteratively, and this agent cannot. I’ll measure what that costs in Part 2 rather than hand-wave it.

The verdict is reported, never acted on. The quarantined model’s output is an untrusted value whose only legal destination is the report. It cannot be a branch condition and it cannot reach an egress sink. So the agent does not close cases, does not escalate and does not prioritize.

Put those together and the guarantee fits in a sentence: by the time the attacker gets to say anything, the program is already fixed, and their only remaining influence is over text that a human reads.

The obvious objection is that this agent doesn’t do very much. My answer is that it does the expensive part — collecting, correlating, enriching, and showing its work — and leaves the decision where the accountability already was. If that sounds like a downgrade, it’s worth asking what the alternative was actually buying you, given the first half of this post.

What This Costs You

I want this on the record before Part 2, so that the numbers arrive as a measurement rather than a sales pitch.

Architectural defenses have a utility tax. The CaMeL paper reports solving a smaller fraction of tasks with provable security than the same system undefended — the security is real, and some tasks stop being expressible. That is the honest shape of this tradeoff, and any writeup that doesn’t quote a cost figure is selling something.

There are three more costs that get discussed even less:

• Latency and tokens. Two model calls where there was one, plus interpretation.
• Human attention. If the design generates twenty approvals per case it is unusable, and the failure will look like adoption failure rather than security failure. I’ve found nobody publishing this number, so Part 2 will.
• Expressiveness. No replanning, no acting on the verdict. Some workflows simply don’t fit, and the correct response is to notice that before building.

And one thing that this design does not buy you, which I’d rather say now: it does not make the report trustworthy. The attacker still controls text that a human is going to read. Everything above bounds what the system does; it does nothing about what the analyst is persuaded to believe. That’s a different problem and I don’t want to imply it’s solved.

What’s Next

Part 2 builds it: the restricted plan language, the label lattice, the interpreter that propagates labels and enforces the four policies, and the quarantined model with schema-constrained output. Then it ports the whole design to LangGraph, where it turns out the framework’s most-recommended agent pattern violates P1 by construction.

Then it attacks it. The attack suite injects through the channels a real triage agent actually reads — log fields, phishing bodies under triage, enrichment responses, EDR telemetry — plus adapted canonical attacks for comparability. And it runs against three models: a frontier model and two open-weights models of different sizes.

That last choice is the whole experiment. If the design is right, the attack success rate under defense should be roughly flat across all three, while benign utility scales with model capability. Flat means the interpreter is doing the defending and the model isn’t. If it isn’t flat, that’s a result too, and it goes in the post exactly as measured.

References

Verified against the arXiv API on 22 August 2026: identifier, title, date and authorship checked. Venue is stated only where the metadata confirms it — which, for these, it does not.

Architectural defenses

Not a paper, and the ancestor of all of the above: Simon Willison, The Dual LLM pattern for building AI assistants that can resist prompt injection, 25 April 2023 — simonwillison.net.

Evaluation and adaptive attack

Injection against log analysis and security operations

This line of work is what makes the Part 2 attack corpus something other than my own invention.