Certified Multi-Source Integrity for Structured Agent Actions

Anmol Pandey, Aditya Jain, Liang Chen, carsten maple, Christo Panchev

TL;DR

LLM agents executing high-stakes structured actions (like processing financial invoices) are highly vulnerable to indirect prompt injection via corrupted data sources. This paper introduces a certified admission gate that isolates evidence sources and aggregates them based on independent control domains rather than raw document counts. The system completely eliminates unsafe actions across multiple commercial LLMs while preserving the agent's ability to process legitimate, novel transactions.

What Problem Does This Paper Address?

As Large Language Model (LLM) agents shift from generating text to executing autonomous, irreversible actions—such as paying invoices, screening counterparties, or updating databases—security risks escalate. A primary threat is indirect prompt injection, where an adversary embeds malicious instructions inside an untrusted document (e.g., an invoice or a third-party data feed). When the LLM reads this document, the injection manipulates the model into extracting attacker-chosen values, leading to fraudulent actions like transferring money to a malicious account.

Current defenses fall short: information-flow tracking and capability gating often enforce a blanket refusal on any untrusted data, destroying the agent's utility. Meanwhile, existing certified robustness techniques focus on free-text answers and fail to handle structured, multi-field actions. Crucially, prior work treats individual documents as independent witnesses, ignoring the fact that multiple data sources often share a single upstream point of failure (e.g., different records hosted on the same platform).

Methods

The authors propose a certified multi-source integrity gate that sits between the LLM's data extraction phase and the final action execution.

Figure 1. The certifier as an admission gate. Each source is read in isolation, every action-critical field is decided by the rule its evidence structure supports and counted over corruption-distinct domains, and the action executes only if every field is admitted, otherwise it abstains. An adversary within budget k corrupts at most k control domains, here the tampered invoice.

The core architecture relies on four pillars:

  1. Isolated Extraction: The LLM processes each data source in a completely isolated runtime container, preventing an injection in one document from contaminating the extraction of others.
  2. Trusted Obligation Skeleton: Before reading untrusted data, a pre-defined schema ($\Gamma$) locks down critical parameters (e.g., transaction keys, amount caps), preventing "selection attacks" that steer the agent to an entirely different transaction.
  3. Corruption-Distinct Counting: Instead of counting raw documents, the certifier maps data sources to their underlying control domains. It uses a minimum hitting set algorithm to calculate the minimum number of independent domains an attacker would need to compromise to falsify a fact. This thwarts "data laundering" attacks where an adversary republishes the same corrupted data across multiple channels to simulate a consensus.
  4. Per-Field Certification Rules: Rather than applying a blanket rule, fields are evaluated based on their structure:
  5. Agreement or Escalate: Requires unanimous agreement across independent domains for identity fields.
  6. Unique Threshold: A Byzantine-fault-tolerant voting mechanism for multi-source value fields.
  7. Deterministic Reconciliation: Cross-checks complementary fields (e.g., verifying an invoiced amount against an approved purchase order ceiling).
  8. Trusted Anchor: Gates single-sourced fields (like bank accounts) against strict allowlists.

Datasets

The paper evaluates the real-world feasibility of multi-source corroboration using three distinct datasets:

  • OpenSanctions Collection: 70,966 international sanctions entities used to measure how frequently authorities cross-list targets.
  • Software Supply-Chain Provenance (npm & PyPI): 450 npm packages and 300 PyPI packages analyzed to evaluate independent witnesses for package-to-repository bindings.
  • Live Model-in-the-Loop Harness: A custom evaluation dataset consisting of 80 payment processing episodes containing embedded social-engineering prompt injections.

Key Insights and Ideas

  • The Illusion of Independence: A major finding is that apparent multi-source redundancy often collapses under scrutiny. For instance, software build provenance (SLSA) and repository back-references look like two independent witnesses. However, because both rely on GitHub infrastructure, a single repository compromise invalidates both, reducing three apparent witnesses to just two independent domains.
Figure 2. The apparent third witness to package-source identity collapses on real infrastructure. The registry, the SLSA build provenance, and the repository back-reference look independent, but the provenance is produced by GitHub Actions and the back-reference is GitHub repository content, so both root in one control domain that a single repository compromise forges. Counting corruption-distinct domains leaves two, not three. This is not marginal: 16.4% of 450 npm packages and 32.3% of 300 PyPI packages appear to carry three independent witnesses, and none do once the shared GitHub root collapses, leaving about 90% at exactly two.
  • Safety via Abstention: By separating exact-value tolerance (always getting the right answer) from safety with abstention (getting the right answer or safely refusing to act), the certifier achieves absolute safety even when data redundancy is low.

Key Results

The certifier was tested against five state-of-the-art models (including GPT-5.5, Gemini 3.1 Pro, and Claude Haiku 4.5) subjected to aggressive prompt injection attacks.

  • Naive Agents Fail Completely: Without the certifier, prompt injections successfully fooled four out of the five models 100% of the time, causing the naive agent to execute fraudulent payments in 72% to 100% of cases.
  • Absolute Safety Achieved: The proposed certifier achieved a 0% unsafe execution rate across all models.
  • High Utility vs. Baselines: Unlike traditional capability gating (which blocks all actions, resulting in 0% utility), the certifier maintained 96% to 100% correct execution overall, successfully processing legitimate payments to novel payees that standard allowlists would reject.
Figure 3. Head-to-head on the multi-source action for a model the injection fools (Haiku 4.5). The certifier is the only one of the four that is at once safe (0% unsafe) and high-utility (98%). The naive agent is fully unsafe, capability gating (CaMeL/FIDES) executes nothing, and the policy monitor (PCAS) is 18% unsafe at only 29% utility. The same pattern holds across all five models.

Limitations and Future Work

  • Scarcity of Corroboration: Real-world data is rarely highly redundant. Only 15% of OpenSanctions entities had three or more independent issuing domains, and no software packages reached three distinct domains.
  • Dependency on Curation: The system relies on accurate, pre-mapped metadata linking data sources to their true upstream control domains.
  • Extractor Vulnerabilities: The safety proof assumes that the LLM can accurately extract text from uncorrupted documents; systematic common-mode parser errors across all sources could still pose a risk.

Summary updated

Abstract

LLM agents increasingly take privileged, often irreversible structured actions, such as paying an invoice. They assemble each action from action-critical fields in documents and tool outputs that an adversary can corrupt, and indirect prompt injection can drive the model itself to extract attacker-chosen values. Current defenses gate on a source’s trust label or certify free-text answer quality. None certifies the integrity of a coupled, policy-bound structured action under a corruption budget that accounts for shared upstream sources. We characterize when such an action is safely certifiable and give the maximally live safe certifier. It admits an action only when each field clears the rule its evidence structure supports: a bounded corruption radius over corruption-distinct evidence classes, counted by a minimum hitting set so that re-publishing or laundered copies cannot manufacture quorum, deterministic reconciliation for complementary fields, and a trusted anchor where the evidence leaves a field single-sourced. We formalize two robustness notions, validate each mechanism by ablation, and measure how often the multi-source precondition holds on sanctions designations (70,966 entities) and software supply-chain provenance (450 packages). Under upper-bound proxies, genuine corroboration is a minority phenomenon in both, and naive attestation counting overstates it, since witnesses that look independent collapse to two corruption-distinct domains once shared origin is counted. Across five current models in a real agent loop, a realistic injection fools every model but one and a naive agent then executes the fraudulent action on most attacks. The certifier admits no unsafe action and recovers the correct value where corroboration permits, while action-gating and provenance baselines are broken in every world of our harness.
BibTeX
@inproceedings{pandey2026certified,
  author = {Anmol Pandey and Aditya Jain and Liang Chen and carsten maple and Christo Panchev},
  title = {{Certified Multi-Source Integrity for Structured Agent Actions}},
  booktitle = {UK AI 2026 Asfullpaper},
  year = {2026},
  url = {https://openreview.net/forum?id=ZvbY7nKcCc},
}
Figure