TL;DR
LLM agents executing high-stakes structured actions (like processing financial invoices) are highly vulnerable to indirect prompt injection via corrupted data sources. This paper introduces a certified admission gate that isolates evidence sources and aggregates them based on independent control domains rather than raw document counts. The system completely eliminates unsafe actions across multiple commercial LLMs while preserving the agent's ability to process legitimate, novel transactions.
What Problem Does This Paper Address?
As Large Language Model (LLM) agents shift from generating text to executing autonomous, irreversible actions—such as paying invoices, screening counterparties, or updating databases—security risks escalate. A primary threat is indirect prompt injection, where an adversary embeds malicious instructions inside an untrusted document (e.g., an invoice or a third-party data feed). When the LLM reads this document, the injection manipulates the model into extracting attacker-chosen values, leading to fraudulent actions like transferring money to a malicious account.
Current defenses fall short: information-flow tracking and capability gating often enforce a blanket refusal on any untrusted data, destroying the agent's utility. Meanwhile, existing certified robustness techniques focus on free-text answers and fail to handle structured, multi-field actions. Crucially, prior work treats individual documents as independent witnesses, ignoring the fact that multiple data sources often share a single upstream point of failure (e.g., different records hosted on the same platform).
Methods
The authors propose a certified multi-source integrity gate that sits between the LLM's data extraction phase and the final action execution.
The core architecture relies on four pillars:
- Isolated Extraction: The LLM processes each data source in a completely isolated runtime container, preventing an injection in one document from contaminating the extraction of others.
- Trusted Obligation Skeleton: Before reading untrusted data, a pre-defined schema ($\Gamma$) locks down critical parameters (e.g., transaction keys, amount caps), preventing "selection attacks" that steer the agent to an entirely different transaction.
- Corruption-Distinct Counting: Instead of counting raw documents, the certifier maps data sources to their underlying control domains. It uses a minimum hitting set algorithm to calculate the minimum number of independent domains an attacker would need to compromise to falsify a fact. This thwarts "data laundering" attacks where an adversary republishes the same corrupted data across multiple channels to simulate a consensus.
- Per-Field Certification Rules: Rather than applying a blanket rule, fields are evaluated based on their structure:
- Agreement or Escalate: Requires unanimous agreement across independent domains for identity fields.
- Unique Threshold: A Byzantine-fault-tolerant voting mechanism for multi-source value fields.
- Deterministic Reconciliation: Cross-checks complementary fields (e.g., verifying an invoiced amount against an approved purchase order ceiling).
- Trusted Anchor: Gates single-sourced fields (like bank accounts) against strict allowlists.
Datasets
The paper evaluates the real-world feasibility of multi-source corroboration using three distinct datasets:
- OpenSanctions Collection: 70,966 international sanctions entities used to measure how frequently authorities cross-list targets.
- Software Supply-Chain Provenance (npm & PyPI): 450 npm packages and 300 PyPI packages analyzed to evaluate independent witnesses for package-to-repository bindings.
- Live Model-in-the-Loop Harness: A custom evaluation dataset consisting of 80 payment processing episodes containing embedded social-engineering prompt injections.
Key Insights and Ideas
- The Illusion of Independence: A major finding is that apparent multi-source redundancy often collapses under scrutiny. For instance, software build provenance (SLSA) and repository back-references look like two independent witnesses. However, because both rely on GitHub infrastructure, a single repository compromise invalidates both, reducing three apparent witnesses to just two independent domains.
- Safety via Abstention: By separating exact-value tolerance (always getting the right answer) from safety with abstention (getting the right answer or safely refusing to act), the certifier achieves absolute safety even when data redundancy is low.
Key Results
The certifier was tested against five state-of-the-art models (including GPT-5.5, Gemini 3.1 Pro, and Claude Haiku 4.5) subjected to aggressive prompt injection attacks.
- Naive Agents Fail Completely: Without the certifier, prompt injections successfully fooled four out of the five models 100% of the time, causing the naive agent to execute fraudulent payments in 72% to 100% of cases.
- Absolute Safety Achieved: The proposed certifier achieved a 0% unsafe execution rate across all models.
- High Utility vs. Baselines: Unlike traditional capability gating (which blocks all actions, resulting in 0% utility), the certifier maintained 96% to 100% correct execution overall, successfully processing legitimate payments to novel payees that standard allowlists would reject.
Limitations and Future Work
- Scarcity of Corroboration: Real-world data is rarely highly redundant. Only 15% of OpenSanctions entities had three or more independent issuing domains, and no software packages reached three distinct domains.
- Dependency on Curation: The system relies on accurate, pre-mapped metadata linking data sources to their true upstream control domains.
- Extractor Vulnerabilities: The safety proof assumes that the LLM can accurately extract text from uncorrupted documents; systematic common-mode parser errors across all sources could still pose a risk.
Summary updated