Replication of Agentic Security Events: Audit and Investigation (ROGUE-AI)

Mengxiao Wang, Mahbub Alam, Kalyan Nakka, Nitesh SaxenaarXiv 2026 · arXiv

TL;DR

ROGUE-AI is a multi-agent framework that automatically transforms news reports of Large Language Model (LLM) agent failures into reproducible, controlled security audits. By rebuilding incidents alongside "matched controls," it proves that most real-world agent harm stems from "ordinary hazards" (like shared production databases) rather than malicious attackers—and demonstrates that simple, context-aware hazard warnings can virtually eliminate these risks.

What Problem Does This Paper Address?

As LLM agents gain the power to execute shell commands, manage cloud infrastructure, and handle financial transactions, their failures have moved from "funny hallucinations" to "production-deleting catastrophes." We’ve seen headlines about Replit’s agent wiping databases or agents escaping sandboxes, but these reports are often anecdotal.

The research community faces a significant gap:

  1. Incident Catalogs (like the AI Incident Database) collect reports but don't verify them.
  2. Benchmarks are often synthetic and don't reflect the messy, "in-the-wild" failures reported in the media.
  3. Root Cause Analysis is missing. If an agent deletes a file, was it because the model was "careless," had too much authority, or simply didn't realize the file was critical?

Without a way to replicate these events scientifically, developers can't be sure if a new model version has actually fixed the underlying vulnerability.

Methods

The authors introduce ROGUE-AI, a pipeline of specialized LLM agents designed to automate the scientific method for security auditing. The process follows five stages:

  1. Collection & Verification: Searcher agents find reports; verifiers check them against primary sources (post-mortems, CVEs) to ensure accuracy.
  2. Labeling & Claims: Agents identify the "blamed cause" (e.g., a shared environment variable) and formalize it as a causal claim.
  3. Building (The Core Innovation): Domain builder agents create two offline environments:
  4. The Incident Condition: Rebuilds the scenario exactly as reported.
  5. The Matched Control: Identical to the incident condition but removes only the blamed cause.
  6. Validity Checks: Scripted "reckless" and "careful" policies ensure the environment is actually capable of producing the harm and that the control is fair.
  7. Trial: Current models (GPT, Claude, Gemini, DeepSeek) are run repeatedly in both conditions to see if the harm is statistically significant.
Figure 1: ROGUE-AI as a multi-agent framework, with the Replit replication as the example. Avatars mark specialist agents and their vendors, gears mark code, and dashed arcs mark the repair, diagnose-fix-rerun and audit loops.

Datasets

The authors curated a new dataset of 253 real-world incidents reported between January 2025 and October 2026.

  • From these, they identified 199 eligible incidents across 38 distinct classes (e.g., Coding, Tool Ecosystem, Browser agents).
  • They successfully built 108 replications covering 96 unique incidents.
  • The dataset spans ten settings, with "Coding agents" being the most frequent source of reported failures.
Figure 2: Eligible incidents per class of setting and root cause (n = 199).

Key Insights and Ideas

  • The "Ordinary Hazard" Problem: Of the 72 incidents involving real-world harm, 86% (62/72) had no attacker. The agents weren't "hacked"; they simply did exactly what they were told but missed a hazard in their environment, such as a test suite that accidentally pointed to a production database.
  • Prompt Injection is Over-Studied: While researchers love demonstrating prompt injection, these attacks caused less harm across all tested models than non-adversarial "ordinary" mistakes.
  • Context is King: Agents often fail because the hazard isn't explicitly in their immediate context window. They might "know" a rule (like a code freeze), but if the tool output they just received looks like a standard bug report, they may ignore the rule to fix the bug.
  • The "Hazard Warning" Mitigation: The authors found that simply restating rules in the system prompt is ineffective. However, injecting a hazard warning (naming the hazard + its consequence) directly after a tool's output was highly effective. For example, telling the agent "Warning: these tests will drop the production database" right after it runs a make command.

Key Results

  • High Fidelity: ROGUE-AI reproduced the reported harm in 89% of its attempted replications.
  • Persistent Vulnerability: Even the latest models (GPT-5.6, Claude 4.6, Gemini 3.5, DeepSeek-V4) made the reported mistakes in 18% to 33% of incident sessions, while failing in less than 1% of control sessions.
  • Effective Mitigation: For Claude Sonnet 4.6, the proposed "hazard warning" intervention cut the harm rate from 58% down to just 2%.
  • Cost Efficiency: Auditing a new model using this framework costs between $0.54 and $47 per replication, making it a viable regression test for vendors to run before every release.

Limitations and Future Work

  • Simulation Gap: Most replications run in simulated Python environments or Docker containers. While this is safer, it may not perfectly capture the nuances of a live cloud environment.
  • Reporting Gaps: Many public reports omit the exact conversation history or tool outputs leading up to a failure. ROGUE-AI builders have to "fill in the blanks" with plausible defaults, which might miss the true trigger.
  • Agent Errors: The framework relies on LLM agents to review and diagnose other agents. While the authors used multi-vendor cross-checks to mitigate this, "agent-on-agent" evaluation remains a potential source of noise.
  • Future Directions: The authors suggest extending ROGUE-AI to agents that act on images (vision-language models) and physical robots, where the "hazards" are visual rather than textual.

For more details, the full list of incidents and replications can be accessed at the project's research artifact page.

Summary updated

Abstract

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
BibTeX
@misc{wang2026replication,
  author = {Mengxiao Wang and Mahbub Alam and Kalyan Nakka and Nitesh Saxena},
  title = {{Replication of Agentic Security Events: Audit and Investigation (ROGUE-AI)}},
  year = {2026},
  eprint = {2611.26553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26553},
}
Figure