Beyond AUROC: Deployment-Aware Evaluation of Model-Internal Jailbreak Detectors

Mohammad Tehabsim, Basil Al-Housani, Nour Alhussien, Phung LaiarXiv 2026 · arXiv

Evaluating jailbreak detectors for Large Language Models (LLMs) often feels like a game of high-stakes whack-a-mole. As researchers move from surface-level filters to "model-internal" detectors that peek at hidden states and gradients, the metrics used to judge them are coming under fire.

This paper argues that the industry-standard metric, AUROC (Area Under the Receiver Operating Characteristic curve), is a misleading "mirage" for real-world security. In deployment, what matters isn't a detector's average performance, but its ability to catch attacks while maintaining a near-zero false-positive rate. The authors demonstrate that current detectors often fail this "deployment-aware" test, revealing massive gaps in coverage that AUROC scores completely hide.

TL;DR

High AUROC scores for jailbreak detectors are often a mirage that hides "operating-point failures" in real-world settings. This paper shows that when detectors are held to strict, deployment-ready false-positive budgets, they frequently miss entire families of attacks, and their performance is dangerously sensitive to the specific data used for calibration and reference.

What Problem Does This Paper Address?

Most evaluations of jailbreak detectors rely on AUROC, which measures how well a detector ranks "harmful" vs. "benign" prompts across all possible thresholds. However, in a production environment, a 5% or 10% false-positive rate (FPR) is unacceptable—it would frustrate users and break legitimate workflows. Security teams need detectors to work at an FPR of 1% or lower.

The authors identify three critical gaps in current research:

  1. The Calibration-Transfer Gap: Thresholds set on "clean" academic datasets (like Dolly) don't work on messy, real-world user traffic (like ToxicChat), leading to far more false alarms than expected.
  2. The Reference-Sensitivity Gap: Many detectors rely on a "reference set" of benign and malicious examples to calculate scores. The authors find that the choice of these examples isn't just an implementation detail—it's a causal factor that changes which attacks the detector can see.
  3. The Operating-Point Gap: A detector can have a stellar AUROC of 0.95 but still have a 0% detection rate at the specific low-FPR threshold required for deployment.

Methods

The researchers formalized jailbreak detection as a data-dependent decision problem. They evaluated four prominent model-internal detectors (MTK, GradSafe, HiddenDetect, and supervised activation probes) across four LLMs (Llama-2, Llama-3, Mistral, and Vicuna).

Key methodological innovations include:

  • Calibration Transfer Analysis: They set thresholds on curated data and measured the "realized FPR" on independent real-user distributions.
  • Causal Reference Interventions: For detectors like MTK that use "anchors," the authors performed controlled experiments—swapping out specific reference anchors to see if it directly caused the detector to miss or catch certain attack families.
  • Per-Family TPR: Instead of aggregate scores, they reported the True Positive Rate (TPR) for 12 distinct jailbreak families at explicit 1% and 5% FPR budgets.

Datasets

  • Benign/Calibration: Dolly (curated instructions), ToxicChat (real-user prompts), WildChat (real-user prompts), and OR-Bench (difficult benign prompts).
  • Attacks: 12 jailbreak families, including suffix-based attacks (SAA, nanoGCG), multilingual (Zulu), and automated templates (AutoDAN, PAIR).
  • New Benchmark: The authors introduced SAA-850, a more diverse version of the Suffix-based Adversarial Attack dataset, to test if detectors generalize across different harmful requests rather than just memorizing specific adversarial suffixes.

Key Insights and Ideas

  • The AUROC Mirage: Figure 1 illustrates the core problem. A detector might look perfect on paper, but when you move the threshold to a real-user distribution, the "detection zone" shifts, and attack coverage can plummet from 100% to near zero.
Fig. 1. Calibration transfer and attack coverage under curated and real-user thresholds.
  • Calibration is Fragile: A threshold that yields a 5% FPR on curated data can explode to a 25% FPR on real-user traffic. This means a "safe" detector in the lab becomes a "blocking machine" in production.
  • Reference Data is a Security Parameter: In detectors like MTK, the reference anchors define the "geometry" of what is considered malicious. The authors found that "improving coverage of one attack family can reduce coverage of others." It’s a zero-sum game that requires careful auditing of the reference set.
  • High AUROC $\neq$ High Security: As shown in Figure 3, several attack families exhibit "sharp changes in TPR over a narrow range of false-positive rates." If the attack scores are clustered just below the deployment threshold, the AUROC stays high, but the actual security value is zero.
Fig. 3. MTK family-level detection versus realized FPR under real-user calibration. Gray curves denote the remaining attack families; the dashed line marks the nominal 5% operating point.

Key Results

  • FPR Explosion: Thresholds calibrated on Dolly exceeded the 5% target by 2.3x to 5.4x when tested on real-user traffic across all models.
  • The 1% Failure: At a strict 1% FPR budget, no evaluated internal detector exceeded a mean family-level TPR of 0.416, despite some having mean AUROCs as high as 0.962.
  • Specific Gaps: On Llama-3, the MTK detector achieved an AUROC of 0.918 for SAA attacks but had a TPR of only 0.002 at the 5% FPR operating point. It was effectively blind to the attack at a usable threshold.

Limitations and Future Work

The authors acknowledge several limitations:

  • Proxy Data: While ToxicChat and WildChat are better than curated sets, they are still only proxies for the infinite variety of real-world deployment environments.
  • Non-Adaptive Attacks: This study focused on known, static attacks. An adaptive attacker who knows the detector's reference set could likely bypass these monitors even more easily.
  • Generalization: The "formatting artifact" issue (where specific data quirks influenced detection) was primarily demonstrated on Llama-2, and its prevalence in other models remains an open question.

The paper concludes with a strong recommendation: stop reporting AUROC as the primary metric. Instead, evaluate detectors using realized FPR on deployment-like traffic and per-family TPR at explicit low-FPR operating points.

Summary updated

Abstract

Large language models (LLMs) are increasingly deployed in user-facing systems, where jailbreak attacks can bypass safety controls and induce harmful and policy-violating outputs. Recent model-internal detectors use hidden representations, gradients, or internal signals to identify such prompts, but their practical security value depends critically on behavior at low false-positive operating points rather than on area under the receiver operating characteristic curve (AUROC) alone. We study two data dependencies that shape this behavior: calibration data, which determine the detection threshold, and reference data, which can influence detector scores. Across four model-internal detectors, four LLMs, and twelve jailbreak families, thresholds calibrated on curated benign prompts frequently fail to preserve their false-positive rate (FPR) budgets on real-user traffic. Recalibration improves false-positive control but reveals substantial family-level coverage gaps, including cases with high AUROC but low true-positive rate (TPR). Causal reference-set interventions show that changing reference composition can substantially alter family-level detection and that improving coverage of one attack family can reduce coverage of others. These findings show that calibration and reference data are part of a detector's security behavior. We therefore recommend evaluating model-internal jailbreak defenses using realized FPR on deployment-like traffic and per-family TPR at explicit low-FPR operating points, with AUROC used as a complementary ranking metric rather than the primary security measure.
BibTeX
@misc{tehabsim2026beyond,
  author = {Mohammad Tehabsim and Basil Al-Housani and Nour Alhussien and Phung Lai},
  title = {{Beyond AUROC: Deployment-Aware Evaluation of Model-Internal Jailbreak Detectors}},
  year = {2026},
  eprint = {2611.11067},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11067},
}
Figure