What Standard Evaluations Hide About Detectors of Hijacked LLM Agents: Rare Attacks, Unseen Tools and Benchmark Transfer

Mahmoud Abbasi, Juan M. Núñez V., Amin ShahrakiarXiv 2026 · arXiv

TL;DR

Standard evaluations for LLM hijack detectors are dangerously optimistic. This paper reveals that high performance on balanced benchmarks evaporates in the real world, where attacks are rare, tools are new, and detectors often rely on "shortcuts" rather than actual behavioral understanding.

What Problem Does This Paper Address?

As we give Large Language Models (LLMs) more agency—allowing them to browse the web, send emails, or access file systems—we open the door to indirect prompt injection. An attacker can plant malicious instructions in a tool's output (like a webpage or an email) to hijack the agent's behavior.

To counter this, researchers have built "trace detectors" that analyze tool-call logs to spot compromised sessions. However, the authors audited 11 recent evaluations and found a massive gap between lab results and deployment reality. Most evaluations use "balanced" test sets (where up to 86% of samples are attacks), ignore the "distribution shift" caused by new tools, and rarely test if a detector trained on one benchmark works on another.

Methods

The authors didn't just propose a new detector; they audited the evaluation methodology itself. They implemented several classes of detectors:

  • Content-blind: Logistic Regression (LR) and Gated Recurrent Units (GRU) that look at tool names and behavioral metadata (e.g., observation length).
  • Content-aware: TF-IDF models that analyze the actual text of tool outputs.
  • Step Embeddings (SE): A reimplementation of the DriftNet architecture using frozen sentence encoders.
  • Untrained Monitors: Using LLMs (like Llama 3.3 or Claude 3.5) to audit traces without specific fine-tuning.

They introduced a novel evaluation design (see Figure 1) that tests for three critical real-world shifts:

  1. Rare Attacks: Testing precision when attacks make up only 1% to 10% of traffic.
  2. Unseen Tools: Gradually masking tool names to simulate an agent encountering tools it wasn't trained on ("graded coverage").
  3. Benchmark Transfer: Training on one dataset and testing on a completely different one.

![Fig.

  1. Detection setting and evaluation design. (a) The audit process. (b) The graded shift settings from standard splits (R1) to cross-benchmark transfer (R4).](/figures/art_a82ee7a2b19f4965955b5892780b217c)

Datasets

The study utilizes two primary benchmarks:

  • AgentDrift: A synthetic dataset of ~12,500 trajectories across domains like banking and medical, generated by Llama-3.
  • AgentDojo: A more diverse benchmark featuring recorded runs from various high-end models (GPT-4o, Claude 3.5) performing tasks in workspace, travel, and Slack environments.

Key Insights and Ideas

  • The Base-Rate Fallacy: A high AUROC (Area Under the ROC Curve) is the standard metric, but it's misleading for security. If attacks are rare (e.g., 2%), a detector with a "great" AUROC of 0.88 might still bury security analysts in false positives, yielding a precision of only 7.6%.
  • Ranking Reversals: Detectors are brittle when toolsets change. As "tool coverage" (the share of known tools) drops below 50%, the rankings of which detector is "best" actually reverse. Sequence models like GRUs fail spectacularly on unseen tools because they encounter "unknown" tokens they never learned to handle.
  • Shortcut Learning: Many detectors aren't learning "hijacked behavior"; they are learning the "surface texture" of the benchmark. For instance, on AgentDrift, detectors often just learned to recognize the specific text templates used by the synthetic generator.
  • The Transfer Gap: Detectors trained on synthetic data (AgentDrift) performed no better than chance when tested on the more realistic AgentDojo. This suggests that current training sets are not capturing the universal "essence" of a hijack.

Key Results

  • Precision Collapse: At a 2% attack rate, most detectors produced over 12 false alerts for every one true hijack. Only a text-based TF-IDF model maintained high precision, but only because it learned specific "attack strings" unique to the benchmark.
  • The Coverage Cliff: As tool coverage falls, the False Positive Rate (FPR) for standard detectors like LR-C skyrocketed from 0.19 to 0.65.
  • Transfer Failure: No detector trained on AgentDrift achieved an AUROC significantly better than 0.50 on AgentDojo.
  • LLM Monitors: The only approach that transferred well was using an untrained LLM as a monitor (AlignmentCheck), which reached an AUROC of 0.93 on AgentDojo. However, even this "smart" monitor would still raise mostly false alerts at a 2% attack rate.

![Fig.

  1. Detection quality falls as attack prevalence drops. (a) wAUPRC and (b) Precision at the training threshold.](/figures/art_fac6b1f975564cd3869c5ea4d32343b0)

Limitations and Future Work

The authors acknowledge that AgentDrift is synthetic and perhaps too easily "solved" by surface cues. Furthermore, AgentDojo has a relatively small pool of benign traces, which can make evaluation noisy.

The paper concludes with a checklist for future researchers, urging them to report precision at low prevalence (1-10%), document tool coverage, and always include a "within-target" control to prove that their detector is learning behavior rather than just benchmark-specific shortcuts.

Summary updated

Abstract

Detectors can flag hijacked large language model (LLM) agents from the traces of the tools they call. In the eleven evaluations we audited, attacks or anomalies make up 16% to 86% of the test samples, no evaluation varies that share, and six report no test on data from a second benchmark or source. We ask what such evaluations hide. On the AgentDrift and AgentDojo benchmarks we test trained detectors that read tool identities, behavior, call sequences, step embeddings or text, and a released LLM action monitor, AlignmentCheck, with thresholds fixed on training data or set by the monitor's released decision rule. First, at a 2% attack rate a detector with an area under the receiver operating characteristic curve (AUROC) of 0.879 reaches a precision of 0.076, and the mix of negatives alone moves that precision between 0.039 and 0.117. Second, as tool coverage, the share of test tool calls whose tool was seen in training, falls, detector rankings cross near 50% coverage and reverse below it, because unseen tools map to inputs that training never reached. Dropout on tool inputs largely removes the reversal. Third, no detector trained on AgentDrift ranks AgentDojo traces better than chance (AUROC 0.47 to 0.52), including one that reaches 0.996 within AgentDojo. The monitor, trained on neither, reaches 0.930, but with its open backbone its released decision rule would still mostly raise false alerts at a 2% attack rate (precision 0.096). We turn these findings into a checklist for evaluating trace detectors.
BibTeX
@misc{abbasi2026what,
  author = {Mahmoud Abbasi and Juan M. Núñez V. and Amin Shahraki},
  title = {{What Standard Evaluations Hide About Detectors of Hijacked LLM Agents: Rare Attacks, Unseen Tools and Benchmark Transfer}},
  year = {2026},
  eprint = {2610.41803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41803},
}
Figure