TL;DR
ROGUE-AI is a multi-agent framework that automatically transforms news reports of Large Language Model (LLM) agent failures into reproducible, controlled security audits. By rebuilding incidents alongside "matched controls," it proves that most real-world agent harm stems from "ordinary hazards" (like shared production databases) rather than malicious attackers—and demonstrates that simple, context-aware hazard warnings can virtually eliminate these risks.
What Problem Does This Paper Address?
As LLM agents gain the power to execute shell commands, manage cloud infrastructure, and handle financial transactions, their failures have moved from "funny hallucinations" to "production-deleting catastrophes." We’ve seen headlines about Replit’s agent wiping databases or agents escaping sandboxes, but these reports are often anecdotal.
The research community faces a significant gap:
- Incident Catalogs (like the AI Incident Database) collect reports but don't verify them.
- Benchmarks are often synthetic and don't reflect the messy, "in-the-wild" failures reported in the media.
- Root Cause Analysis is missing. If an agent deletes a file, was it because the model was "careless," had too much authority, or simply didn't realize the file was critical?
Without a way to replicate these events scientifically, developers can't be sure if a new model version has actually fixed the underlying vulnerability.
Methods
The authors introduce ROGUE-AI, a pipeline of specialized LLM agents designed to automate the scientific method for security auditing. The process follows five stages:
- Collection & Verification: Searcher agents find reports; verifiers check them against primary sources (post-mortems, CVEs) to ensure accuracy.
- Labeling & Claims: Agents identify the "blamed cause" (e.g., a shared environment variable) and formalize it as a causal claim.
- Building (The Core Innovation): Domain builder agents create two offline environments:
- The Incident Condition: Rebuilds the scenario exactly as reported.
- The Matched Control: Identical to the incident condition but removes only the blamed cause.
- Validity Checks: Scripted "reckless" and "careful" policies ensure the environment is actually capable of producing the harm and that the control is fair.
- Trial: Current models (GPT, Claude, Gemini, DeepSeek) are run repeatedly in both conditions to see if the harm is statistically significant.
Datasets
The authors curated a new dataset of 253 real-world incidents reported between January 2025 and October 2026.
- From these, they identified 199 eligible incidents across 38 distinct classes (e.g., Coding, Tool Ecosystem, Browser agents).
- They successfully built 108 replications covering 96 unique incidents.
- The dataset spans ten settings, with "Coding agents" being the most frequent source of reported failures.
Key Insights and Ideas
- The "Ordinary Hazard" Problem: Of the 72 incidents involving real-world harm, 86% (62/72) had no attacker. The agents weren't "hacked"; they simply did exactly what they were told but missed a hazard in their environment, such as a test suite that accidentally pointed to a production database.
- Prompt Injection is Over-Studied: While researchers love demonstrating prompt injection, these attacks caused less harm across all tested models than non-adversarial "ordinary" mistakes.
- Context is King: Agents often fail because the hazard isn't explicitly in their immediate context window. They might "know" a rule (like a code freeze), but if the tool output they just received looks like a standard bug report, they may ignore the rule to fix the bug.
- The "Hazard Warning" Mitigation: The authors found that simply restating rules in the system prompt is ineffective. However, injecting a hazard warning (naming the hazard + its consequence) directly after a tool's output was highly effective. For example, telling the agent "Warning: these tests will drop the production database" right after it runs a
makecommand.
Key Results
- High Fidelity: ROGUE-AI reproduced the reported harm in 89% of its attempted replications.
- Persistent Vulnerability: Even the latest models (GPT-5.6, Claude 4.6, Gemini 3.5, DeepSeek-V4) made the reported mistakes in 18% to 33% of incident sessions, while failing in less than 1% of control sessions.
- Effective Mitigation: For Claude Sonnet 4.6, the proposed "hazard warning" intervention cut the harm rate from 58% down to just 2%.
- Cost Efficiency: Auditing a new model using this framework costs between $0.54 and $47 per replication, making it a viable regression test for vendors to run before every release.
Limitations and Future Work
- Simulation Gap: Most replications run in simulated Python environments or Docker containers. While this is safer, it may not perfectly capture the nuances of a live cloud environment.
- Reporting Gaps: Many public reports omit the exact conversation history or tool outputs leading up to a failure. ROGUE-AI builders have to "fill in the blanks" with plausible defaults, which might miss the true trigger.
- Agent Errors: The framework relies on LLM agents to review and diagnose other agents. While the authors used multi-vendor cross-checks to mitigate this, "agent-on-agent" evaluation remains a potential source of noise.
- Future Directions: The authors suggest extending ROGUE-AI to agents that act on images (vision-language models) and physical robots, where the "hazards" are visual rather than textual.
For more details, the full list of incidents and replications can be accessed at the project's research artifact page.
Summary updated