I Built the Defense, Then Attacked It: Numbers From a Secure Triage Agent

Table of Contents

TL;DR

• I built the design from Part 1: a privileged planner emitting a restricted plan, a quarantined model with no tool access, and an interpreter that propagates labels and enforces four policies deterministically.
• Then I attacked it: 20 injection cases across five channels a real triage agent reads, 8 clean alerts, five configurations, three models — gpt-5.4, qwen2.5:14b and qwen2.5:7b. 420 runs.
• The policy-enforced objectives go to zero and stay there. Exfiltration undefended runs 3/5, 1/5, 4/5 across the three models; with the data-flow policy on it is 0/5, 0/5, 0/5. Disclosure the same. Budget exhaustion the same. Flat, because on those paths no model is consulted about anything.
• Evasion is the opposite. Undefended it tracks model strength almost perfectly: 1/5 on the frontier model, 4/5 at 14B, 5/5 at 7B. The architecture cuts it hard (5/5 → 2/5) but cannot remove it, and what’s left scales with capability.
• The detector row is a disaster on a weak model. On qwen2.5:7b it flagged 20 of 20 attacked runs, still let 5 attacks through, and dropped utility under attack to 1/20. The interpreter on the same model: 16/20.
• Two things I got wrong. Structure without the data-flow policy is worse than no structure. And one of my four policies was pure friction — it fired up to 3.9 times per alert and declined nothing across all 420 runs.
• There is a LangGraph port in here too, because it makes the argument faster than the interpreter does: create_react_agent and its conditional edge violate P1 by construction, in twelve lines of code everybody has already written.

The Agent and the Corpus

The agent triages a SOC alert: read it, pull the raw log and any reported mail body, extract indicators, enrich them against a reputation service, produce a verdict. It is read-only, so the consequential act is disclosure — you cannot un-submit a hash, and a lookup against an attacker-controlled domain carries the internal hostname out in the subdomain.

The corpus is 8 clean alerts with ground-truth verdicts and 20 attack cases: five channels crossed with four objectives. Corpus size was fixed before the first run, which matters — deciding n after seeing results is the methodological failure I spent a whole post criticising.

Channels 1 and 3 aren’t invented. Injection through log content against LLM-based interpretation is a documented line of work — see the references.

The environment is simulated end to end. No lookup leaves the machine, which removes network variance and also removes any chance of my attack suite doing something impolite to a live reputation service.

Where This Comes From

Almost nothing about the architecture below is mine, and you should know which parts are load-bearing literature before you read the code — partly because credit, and partly because the borrowed parts have been attacked by people better at it than me, which is the main reason to trust them.

The lineage is four steps long:

Simon Willison’s Dual LLM pattern, 25 April 2023. A privileged LLM that can call tools but never sees untrusted content, and a quarantined LLM that reads untrusted content but cannot act. The words privileged and quarantine in the code below are his, including the quarantine.classify in the plan itself.

CaMeL (arXiv:2503.18813) turned the pattern into a mechanism: extract the control and data flow from the trusted query, so untrusted data can never affect the program flow, and attach capabilities to values so policy can be checked when tools are called. That is the design of this post, and the paper states its price — 77% of AgentDojo tasks solved with provable security against 84% undefended.

Fides (arXiv:2505.23643) takes the information-flow route directly: a planner carrying confidentiality and integrity labels, with explicit declassification. The Label class below is that idea with the serial numbers still on it.

Beurer-Kellner et al. (arXiv:2506.08837) name six patterns that trade agent generality for resistance. The topology here is Plan-Then-Execute layered over Dual LLM — so the right-hand side of the LangGraph diagram later in this post is not something I came up with, it is a pattern with a name and a paper.

What I changed, and why

The useful part of an attribution section is the diff, so here it is.

So the honest summary of the contribution: I took an existing architecture, specialised it to a domain where the declassification rule can be structural rather than learned, and measured the two things the papers do not — what happens to the guarantee when you swap a frontier model for a 7B one, and what the design costs in human approvals per alert.

The architecture is theirs. The numbers are mine, and so are the three mistakes in the section near the end.

Four Pieces, and What Each One Is For

The next four sections build the machine. They land better if you know what each part is for before you meet it, so here is the whole design in one table. There are exactly four moving parts — a small custom language, labels on values, a gatekeeper and a runner — and each one exists because removing it breaks a named property:

Two things about that table are worth holding onto, because they are the non-obvious part of the whole design.

The piece carrying the weight is not the one you’d guess. The interpreter sounds like the security component and is actually mechanical bookkeeping — it does what the labels tell it. Every judgment call in the system lives in the declassifier, and it is the only component where being wrong is silent.

Nothing in that column says “detect”. No piece inspects text for malicious intent. That is the entire difference from the previous post: these components decide what a value is allowed to do based on where it came from, never on what it appears to say.

The Plan DSL

DSL is domain-specific language: a small language built to express one kind of thing and nothing else, the way SQL expresses queries and cannot be talked into expressing anything else. Here that refusal is the whole security argument. The plan language below is a subset of Python large enough to call tools, bind names and walk lists, and too small to say if. So “never branch on untrusted data” is not a rule the system checks at runtime — it is a sentence the language cannot form.

The privileged model sees only the trusted instruction — the alert ID and the rule name. Never the log. It emits this:

alert = siem.get_alert(ALERT_ID)
raw = siem.get_raw_log(alert.log_ref)
body = mail.get_body(alert.mail_ref)
cands = util.concat(quarantine.extract_indicators(raw),
                    quarantine.extract_indicators(body))
safe = [policy.declassify(c) for c in cands]
reps = [ti.lookup(s) for s in safe]
verdict = quarantine.classify(alert.rule, reps)
report.emit(verdict, reps)

The grammar is deliberately tiny:

ALLOWED_NODES = {
    ast.Module, ast.Expr, ast.Assign, ast.Name, ast.Load, ast.Store,
    ast.Call, ast.Attribute, ast.Constant, ast.List, ast.Tuple,
    ast.ListComp, ast.comprehension, ast.keyword,
}

Read that by what’s missing. No If, no While, no IfExp — so “never branch on untrusted data” holds by construction, not by a check that could have a bug. No Import, no FunctionDef. And no Subscript, which is less obvious and turns out to matter: with no way to select an element by position, and with comprehension filters rejected, which element you get can never encode information. That is precisely what makes per-element declassification sound later on.

def validate_plan(src: str) -> ast.Module:
    tree = ast.parse(src, mode="exec")
    for node in ast.walk(tree):
        if type(node) not in ALLOWED_NODES:
            raise PlanRejected(f"disallowed construct {type(node).__name__}")
        if isinstance(node, ast.Attribute) and node.attr.startswith("_"):
            raise PlanRejected("private attribute access is not permitted")
    return tree

The whole plan is validated before the first tool call. Nothing is re-planned after untrusted data arrives.

Worth noting for anyone assuming this needs a big model: qwen2.5:7b writes a valid plan in this grammar, first try, and got the verdict right on the smoke case. The grammar is small enough that planning is not the hard part.

The Label Lattice

Every runtime value carries three things: can I believe it, who may see it, and where did it come from.

class Integrity(IntEnum):
    UNTRUSTED = 0
    TRUSTED = 1

class Conf(IntEnum):
    PUBLIC = 0
    INTERNAL = 1
    RESTRICTED = 2

@dataclass(frozen=True)
class Label:
    integrity: Integrity = Integrity.TRUSTED
    conf: Conf = Conf.PUBLIC
    prov: frozenset = frozenset()

    def join(self, other: "Label") -> "Label":
        """Combining two values takes the worst of each dimension."""
        return Label(
            integrity=Integrity(min(self.integrity, other.integrity)),
            conf=Conf(max(self.conf, other.conf)),
            prov=self.prov | other.prov,
        )

Integrity takes the minimum, confidentiality the maximum, provenance the union. Standard, and the entire enforcement story rests on it.

The Declassifier Is the Whole Ballgame

Here is the problem nobody warns you about. Indicators extracted from a log inherit INTERNAL from the log. The reputation service is an egress sink that requires PUBLIC. So with a naive information-flow policy, every enrichment is blocked and the agent is useless. That is over-tainting, and it is the real cost of this approach.

The fix is an explicit declassification step — and the crucial property is that it is deterministic, structural, and has no model in it:

def declassify_indicator(ind: str) -> tuple[bool, str]:
    ind = ind.strip().lower()

    if IPV4_RE.match(ind):
        addr = ipaddress.ip_address(ind)
        if any(addr in net for net in INTERNAL_NETS):
            return False, "internal-ipv4"
        return True, "external-ipv4"

    if SHA256_RE.match(ind):
        if ind in INTERNAL_ARTIFACTS:
            return False, "internal-artifact-hash"
        return True, "sha256"

    if DOMAIN_RE.match(ind):
        labels = ind.split(".")
        for lb in labels:
            if lb in INTERNAL_HOSTS:
                return False, f"label-names-internal-host:{lb}"
            for host in INTERNAL_HOSTS:
                if host in lb:
                    return False, f"label-contains-internal-host:{host}"
        for dom in INTERNAL_DOMAINS:
            if ind == dom or ind.endswith("." + dom):
                return False, "internal-domain"
        for lb in labels[:-2]:
            if len(lb) >= 16 and _entropy(lb) > 3.5:
                return False, f"high-entropy-label:{lb[:12]}..."
        return True, "public-domain"

    return False, "unrecognised-indicator-format"

This is what kills the exfiltration attack. The payload asks the agent to resolve web01-prod.collect.attacker-cdn.example; the check sees a label naming an internal host and refuses to clear it for egress. Confidentiality stays INTERNAL, the sink requires PUBLIC, the call is refused.

Note what declassification does not do: it lowers confidentiality, never integrity. Deciding a value may be disclosed says nothing about whether it can be believed.

And note the last check honestly. The inventory and network checks are structural — they cannot be argued with. The entropy check is a heuristic, and by the argument of the previous post a heuristic is not a security property. It is there to raise the cost of encoded exfiltration, not to prevent it. I come back to this in the failure section.

There is a flaw visible in that capture and it is mine. The extractor hands the declassifier powershell.exe and winword.exe, and the domain rule waves them through, because .exe parses as a top-level domain under [a-zA-Z]{2,63}. Harmless for the objectives measured here — neither is an internal hostname, and the attack criteria key on the hostname and the artifact hash — but it means process names reach a third party, which is the quiet kind of disclosure this policy exists to stop. The indicator grammar needs a real TLD allowlist, not a letter run. I found it by reading my own trace while writing this section, which is an argument for printing the trace.

The Interpreter and the Four Policies

def check_policies(self, spec, args, arg_label):
    if self.enforce_p4:
        if self.budget.calls >= self.budget.max_calls:
            raise PolicyViolation("P4", "call budget exhausted")
        if spec.is_egress and self.budget.egress >= self.budget.max_egress:
            raise PolicyViolation("P4", "egress budget exhausted")

    if self.enforce_p1 and not spec.accepts_untrusted:
        if arg_label.integrity == Integrity.UNTRUSTED:
            raise PolicyViolation("P1", f"{spec.name} rejects untrusted arguments")

    if self.enforce_p2 and spec.is_egress:
        if arg_label.conf > spec.max_conf_out:
            raise PolicyViolation("P2", f"{spec.name} limited to {spec.max_conf_out.name}")

    if self.enforce_p3 and spec.is_egress:
        ok = self.approve(desc, arg_label)
        if not ok:
            raise PolicyViolation("P3", f"human declined: {desc}")

One decision worth calling out, because I got it wrong first: a violation must not always abort the plan. P1 and P4 are structural — the plan is wrong, or the run is over — so they abort. P2 and P3 are per-value judgments, so the call is refused and execution continues:

try:
    self.check_policies(spec, all_args, arg_label)
except PolicyViolation as v:
    if v.policy in ("P1", "P4"):
        raise
    self.trace.violations.append({"tool": spec.name, "policy": v.policy})
    return Labeled({"status": f"refused:{v.policy}"}, TRUSTED)

My first version aborted on any violation. That hands the attacker a denial of service: plant one poisoned indicator and the whole triage dies.

There was a subtler bug too. Iterating a list originally joined each element’s label with the container’s, which silently undid every declassification — safe was a list of PUBLIC elements inside an INTERNAL container, so reading an element gave you INTERNAL right back and the agent could enrich nothing. Binding the element’s own label is correct here only because the grammar has no filters and no indexing. Remove that restriction and container taint becomes necessary again.

Building This in LangGraph

Everything above is a from-scratch interpreter. That is the honest way to explain a mechanism and it is not how anyone ships. In practice this gets built on a framework, so here is the port — including the part where the framework’s most-recommended pattern is the vulnerability.

The idiom that breaks P1

This is the standard agent loop. Some version of it is in every LangGraph tutorial, including mine:

def should_continue(state: MessagesState) -> str:
    last = state["messages"][-1]
    return "tools" if last.tool_calls else END

builder.add_conditional_edges("agent", should_continue, ["tools", END])

Read it against P1. add_conditional_edges is a branch. Its condition is last, which came out of a model whose context contains the raw log. So a sentence the attacker wrote into that log selects which edge the graph takes — and selects it again on every iteration, because tool results feed straight back into the same model that decides.

That is the if verdict == "malicious" violation from Part 1, with two differences that make it worse. It is the pattern everyone is told to start from. And it does not look like a branch on data: it looks like architecture. create_react_agent is the same loop with the edge folded out of sight.

The graph is not the plan

That is the sentence to port. In the ReAct topology the graph is the reasoning — it is redrawn at runtime, one edge at a time, by a model reading untrusted text. In this design the graph is a fixed four-node pipeline that never changes, and the plan is data flowing through it. That topology is Plan-Then-Execute from the design-pattern paper, layered over Willison’s Dual LLM split — so if the right-hand diagram looks like a downgrade from a real agent, that downgrade is precisely the trade the pattern is named for.

Which means the state has an unusual shape for LangGraph. Note what is missing:

class TriageState(TypedDict):
    alert_id: str            # trusted — comes from the analyst
    rule_name: str           # trusted — comes from the SIEM rule set
    plan_src: str            # emitted by the planner, not yet validated
    env: dict                # name -> Labeled, the interpreter's bindings
    trace: list              # every call, its labels, its policy decisions
    report: dict | None

There is no messages. No accumulating transcript, because there is no loop feeding tool output back to a model that decides anything. Losing MessagesState is not a limitation I worked around — it is most of the security property. A message list is a single string channel where labels go to die: once the log text and the instruction are both "content", nothing downstream can tell them apart.

The three zones as nodes

def plan_node(state: TriageState) -> dict:
    resp = planner_llm.invoke([
        SystemMessage(content=PLAN_GRAMMAR_PROMPT),
        HumanMessage(content=f"alert_id={state['alert_id']} rule={state['rule_name']}"),
    ])
    return {"plan_src": resp.content}


def validate_node(state: TriageState) -> dict:
    validate_plan(state["plan_src"])          # raises PlanRejected — fail closed
    return {}


def interpret_node(state: TriageState) -> dict:
    interp = Interpreter(TOOLS, budget=Budget(max_calls=12, max_egress=6))
    report = interp.run(state["plan_src"])
    return {"env": interp.env, "trace": interp.trace, "report": report}

The entire security argument for plan_node is its second message. It is two trusted fields and nothing else. Any change that widens it — “give the planner the log so it can plan better” — collapses the guarantee silently, and it will look like a helpful improvement in code review.

Wiring, in full:

builder = StateGraph(TriageState)
builder.add_node("plan", plan_node)
builder.add_node("validate", validate_node)
builder.add_node("interpret", interpret_node)

builder.add_edge(START, "plan")
builder.add_edge("plan", "validate")
builder.add_edge("validate", "interpret")
builder.add_edge("interpret", END)

graph = builder.compile(checkpointer=InMemorySaver())

Four edges, no add_conditional_edges, no cycle. If while porting this you find yourself reaching for a conditional edge, the question to ask is not “is this convenient” but “what does the condition derive from”. If the answer traces back to anything a tool returned, you have reintroduced the left-hand diagram.

Tools have to declare their own trust properties

This is the piece LangGraph has no opinion about. A @tool gives you a name, a schema and a docstring — none of which say whether calling it tells a third party something, or whether its return value can be believed. So the tool table carries that:

@dataclass(frozen=True)
class ToolSpec:
    fn: Callable
    accepts_untrusted: bool   # may an UNTRUSTED argument reach it?          (P1)
    is_egress: bool           # does calling it tell an outsider something?  (P2/P3)
    max_conf_out: Conf        # highest confidentiality it may receive       (P2)
    out_label: Label          # what its return value is labelled

TOOLS = {
    "siem.get_raw_log": ToolSpec(
        siem_get_raw_log, accepts_untrusted=False, is_egress=False,
        max_conf_out=Conf.RESTRICTED,
        out_label=Label(Integrity.UNTRUSTED, Conf.INTERNAL, {"siem"})),

    "ti.lookup": ToolSpec(
        ti_lookup, accepts_untrusted=True, is_egress=True,
        max_conf_out=Conf.PUBLIC,
        out_label=Label(Integrity.UNTRUSTED, Conf.PUBLIC, {"ti"})),
}

Look at siem.get_raw_log. It is a first-party tool, hitting our own SIEM — the security information and event management platform, the system the whole SOC runs on — with our own credentials — and its output is labelled UNTRUSTED. That is the whole mental shift in one line. The integrity question is never “do I trust this vendor”. It is “can the adversary write into what this returns”, and for a log collector the answer is yes by design: collecting what attackers do is its job. Teams that get this wrong label by vendor and end up trusting the one tool whose entire purpose is to ingest hostile input.

The quarantined model is a tool, not a node

The instinct in LangGraph is to make every model call a node. Here that is exactly wrong:

"quarantine.classify": ToolSpec(
    quarantine_classify, accepts_untrusted=True, is_egress=False,
    max_conf_out=Conf.RESTRICTED,
    out_label=Label(Integrity.UNTRUSTED, Conf.INTERNAL, {"quarantine"})),

It has no tools bound to it, its output is schema-constrained, and its return value is labelled UNTRUSTED — so the interpreter will refuse to let it become a branch condition or reach an egress sink. Its only legal destination is the report. A node sits on the graph’s control path; a tool sits under the interpreter, where the labels apply. Same model call, completely different blast radius.

P3 is the one place the framework genuinely helps

from langgraph.types import interrupt

def approve(desc: str, label: Label) -> bool:
    decision = interrupt({
        "action": desc,
        "provenance": sorted(label.prov),
        "confidentiality": label.conf.name,
    })
    return decision["approved"]

With a checkpointer, that suspends the run durably and resumes it days later on a Command(resume=...). Note what gets handed to the human: the provenance chain, not a summary the model wrote. Part 1 required that and it sounded expensive; interrupt makes it nearly free.

And then apply the lesson from the results below — put this on the override path, where P2 wants to refuse and somebody must take responsibility for overriding it. Wiring it to every egress call is how you get 3.9 prompts per alert and zero declines.

What you get free, and what you still write

Read the right-hand column honestly: LangGraph is a good execution substrate with real durability and real human-in-the-loop primitives, and no opinion whatsoever about information flow. Every component that made the numbers in the next section go flat is a component you write yourself. The framework will not stop you building the vulnerable version, and its documentation will help you do it faster.

Idioms that quietly break a policy

Every row is a thing I either did or nearly did. The last one is the seductive one: the declassifier is fiddly, an LLM would handle the edge cases nicely, and the moment you do it the guarantee stops being a guarantee and goes back to being a probability.

Results

Five configurations × 28 cases × three models = 420 runs. Fractions rather than percentages, because with five cases per objective a percentage pretends to a precision it does not have.

The two open-weights models are the same family at two sizes, which is deliberate: it isolates capability instead of confounding it with a different lab’s training.

The policy-enforced objectives are flat

O2 — exfiltration of an internal hostname

O3 — irreversible disclosure of an internal artifact

O4 — budget exhaustion

This is the result the series was built to test. Undefended, the numbers scatter — 3/5, 1/5, 4/5 — because they depend on whether a given model happens to take the bait. With the policy on, all nine cells are zero. No model is consulted anywhere on those paths, so there is nothing left for model capability to vary.

Evasion is the exact opposite

O1 — verdict flipped to benign

Undefended, this is almost a straight line down the capability axis: the frontier model swallowed the “authorised internal penetration test” story once in five, the 7B swallowed it every single time.

The architecture helps here, and I did not expect it to: 5/5 → 2/5 at 7B, 4/5 → 1/5 at 14B. The mechanism is structural rather than clever — the classifier under the plan sees the enrichment records, not the raw log, so most of the injected text never reaches it. Restricting what flows into the model that decides is itself a mitigation.

But it does not go to zero, and the residual tracks capability. The architecture bounds what the agent does; what it believes is still the model’s problem.

Utility, and what the detector costs

Benign utility on the 8 clean alerts, all four policies on: 8/8, 8/8, 5/8. It scales with capability, as expected — the utility tax is real and it is paid mostly by the small model.

The detector row is there on purpose, and it is worse than I expected. It is not a small BERT — it is the same frontier model prompted to do Prompt Guard’s job, which makes it a stronger baseline than the thing being criticised. It never raised a false positive on a clean alert. And on qwen2.5:7b it flagged all 20 attacked runs, still let 5 attacks through, and left the agent able to complete 1 of 20 tasks, because withholding flagged content from an agent that needed it destroys the job.

On that same model the interpreter scored 16/20 with three of four objectives at zero. That is the comparison the two posts exist to make.

Cost in human attention

Three Things I Got Wrong

1 · Structure without policy is worse than no structure

Look at the P1 only rows. Exfiltration goes 3/5 → 4/5 on the frontier model, 1/5 → 3/5 at 14B, 4/5 → 4/5 at 7B. Budget exhaustion goes from 0/5 undefended to 3/5, 1/5 and 1/5. Adding the interpreter without the data-flow policy made the agent more exploitable, on every model.

The reason is in the enrichment counts: the planned agent performs 4.2 lookups per alert against the undefended agent’s 1.8. A plan says “enrich every indicator you extracted” and then does it, reliably, including the one the attacker planted. The undefended agent wanders, gets bored, and sometimes never looks up the payload at all.

So the sloppiness of an unstructured agent was accidentally protective, and replacing it with discipline while omitting the flow policy is a downgrade. If you are midway through adopting this pattern — plans in, labels not yet — that is the worst place to stop.

2 · One of my four policies was pure friction

P3 asked for human approval on every egress. It fired up to 3.9 times per alert and declined nothing, ever, across all 420 runs.

Obvious in hindsight: by the time P3 runs, P2 has already refused anything carrying internal provenance. Every value P3 sees has been declassified, so the approver has no reason to say no. I specified two controls at the same boundary and only one of them was doing work.

The measurement makes the fix clear. Approval belongs on the override path — ask the human when P2 wants to refuse and somebody needs to take responsibility for overriding it, not as a rubber stamp on values already proven safe. Which is the human-oversight-as-vulnerability contradiction from Part 1, arrived at from the other direction: four prompts an alert trains the analyst to click through, and buys nothing.

3 · “Evasion is immune to the architecture” — true for one model, false in general

I ran gpt-5.4 first, saw 1/5 in all five configurations, and wrote down that no defense in the stack moved evasion at all. On one model that is exactly what the data said.

It was wrong. At 7B the architecture takes evasion from 5/5 to 2/5, and at 14B from 4/5 to 1/5. The frontier model simply resisted the payload well enough undefended that there was no headroom to see the effect.

Which is a small lesson about single-model evaluation, in a post about evaluation: the flat line I measured was a ceiling effect, and I would have published it as a finding.

Where It Still Breaks

Encoded exfiltration. The declassifier’s structural checks catch an internal hostname in a domain label. They do not catch base32 of that hostname. I added an entropy heuristic and it caught my sample, but a heuristic is exactly what Part 1 argued you cannot build a guarantee on. The sound claim is narrower than “no exfiltration”: the channel is reduced to what survives the structural checks, and its bandwidth is capped by P4.

The verdict channel, as measured above — and worse the weaker your model.

The planner itself. I trust it because it only sees the alert ID and the rule name. If any untrusted text ever reaches the privileged model the whole guarantee collapses, and “the rule name is trusted” is an assumption about the SIEM, not something the interpreter enforces.

No adaptive attack. Every number here comes from a fixed corpus written before the defense was finished. By my own argument that systematically overestimates. An attacker who reads this post attacks the declassifier, not the prompt — and that is the obvious next piece of work.

Five cases per objective, single run per cell, no repeats. The zeros are nine independent cells all landing on zero, which is why I believe them; the non-zero numbers are coarse and should be read as direction, not magnitude.

If You Are Building One of These

• The declassifier is the security boundary, not the interpreter. The interpreter is mechanical. All the judgment lives in the function deciding what may cross, and it must be structural — the moment a model decides declassification, you have rebuilt the thing that doesn’t work.
• Do not ship the plan layer without the flow policy. It is worse than what you had, on every model I tested.
• Count your approval prompts, and count the declines. If nobody ever declines, the control is decoration and it is training your analysts to click through.
• Your weak-model results are your real results. Everything that depended on the model looked fine on the frontier model and fell apart at 7B. Everything that depended on the interpreter did not move.
• Decide up front what you are not defending. I am not defending the verdict. Saying so is what keeps the rest of the claims credible.

References

Verified against the arXiv API on 22 August 2026: identifier, title, date and authorship checked. Venue stated only where the metadata confirms it, which for these it does not.

Not a paper, and the ancestor of the whole architecture: Simon Willison, The Dual LLM pattern for building AI assistants that can resist prompt injection, 25 April 2023 — simonwillison.net.