JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework

Fan Liu, Yue Feng, Zhao Xu, Lixin Su, Xinyu Ma, Dawei Yin, Hao LiuSubmission to ICLR 2025 · Rejected

TL;DR

Evaluating whether a large language model (LLM) has actually been compromised by a jailbreak attack is surprisingly difficult—even GPT-4 struggles on subtle, multilingual, or role-playing attacks. JAILJUDGE tackles this with a massive benchmark spanning 14 hazard categories and complex prompts, a Dempster–Shafer evidence-based multi-agent judging framework, and JAILJUDGE Guard, an open-source 7B judge model that delivers interpretable, fine-grained safety scores without API costs.

What Problem Does This Paper Address?

While red-teaming and safety defenses have proliferated, the jailbreak judge—the mechanism used to decide whether an LLM's response actually constitutes a successful jailbreak—remains a major bottleneck:

  1. Heuristics and toxic classifiers fail on context: Keyword matching produces high false-positive rates on benign queries, while toxic classifiers cannot distinguish between a model generating harm and a model politely explaining a refusal.
  2. LLM judges lack explainability and consistency: Prompting GPT-4 or using moderation checkpoints (e.g., Llama Guard, ShieldGemma) often yields contradictory verdicts in complex scenarios (like fictional framing or multi-turn deception), scoring as low as 55% F1.
  3. Multilingual and zero-shot blindness: Existing evaluation tools struggle to generalize across diverse risk categories and exhibit severe evaluation bias in low-resource languages.

Methods

The authors introduce a collaborative judging pipeline and distill it into an open-source guardian:

[Prompt + Response]
        │
        ▼
┌──────────────────┐
│  Judging Agents  │ ──► Scores (1–10) + Initial Reasoning
└──────────────────┘
        │
        ▼  (Dempster–Shafer Evidence Theory Aggregation)
┌──────────────────┐
│  Voting Agents   │ ──► Accept / Reject + Verification
└──────────────────┘
        │
        ▼
┌──────────────────┐
│ Inference Agent  │ ──► Final Jailbreak Score (1–10) + Rationale
└──────────────────┘
  • JailJudge MultiAgent: Rather than querying a single LLM, the framework decomposes evaluation across three agent types:
  • Judging Agents: Evaluate prompt-response pairs and output a 1–10 score alongside explanatory text.
  • Uncertainty-Aware Evidence Aggregation: Scores are converted into Basic Probability Assignments (BPAs) via Dempster–Shafer Theory, explicitly modeling epistemic uncertainty ($\{JB\ \&\ NJB\}$) and pruning communication costs from $O(k \cdot m)$ to $O(1 \cdot m)$.
  • Voting & Inference Agents: Voting agents independently validate or challenge the aggregated score against safety policies, after which an inference agent outputs the final rationale and score.
  • JAILJUDGE Guard: Using the high-quality explanations and scores generated by the multi-agent system, the authors instruction-tune an open-source Llama-2-7B model via LoRA to provide end-to-end, explainable safety evaluations without API fees.
  • Downstream Applications:
  • JailBoost: Uses JAILJUDGE Guard’s 1–10 score as an iterative reward signal to guide adversarial prompt generation.
  • GuardShield: A moderation defense layer that intercepts jailbreak attempts exceeding a score threshold ($\tau_d$) and generates explanatory refusal messages.

Datasets

The authors built the JAILJUDGE benchmark across 14 standardized hazard categories (extending MLCommons with Code Interpreter Abuse) and 5 prompt complexity levels:

  • JAILJUDGETRAIN: 35k+ instruction-tuning pairs combining vanilla, synthetic, in-the-wild, deceptive, and adversarial prompts across diverse proprietary, open, and defended LLM outputs.
  • JAILJUDGETEST (Human-Annotated):
  • JAILJUDGE ID (In-Distribution): 4.5k+ prompt-response pairs verified through multi-stage human and GPT-4 cross-verification.
  • JAILJUDGE OOD (Out-of-Distribution Multilingual): 6k+ labeled pairs across 10 languages spanning high-resource (EN, ZH, IT, VI), medium-resource (AR, KO, TH), and low-resource (BN, SW, JV) languages.
  • External Evaluation Sets: Evaluated on JBB Behaviors (~200 samples), WILDTEST (1,700+ real-world interactions), and HEx-PHI (330 samples for downstream attack/defense evaluation).

Key Insights and Ideas

  • Reasoning grounds the verdict: Requiring structured rationale eliminates “verdict drift,” where judges identify dangerous instructions in their scratchpad but still output a negative jailbreak verdict due to deceptive framing.
  • Evidence theory handles judge disagreement: Instead of raw majority voting, Dempster’s rule of combination accounts for confidence and ambiguity across judging agents, boosting F1 by over 30 points on complex datasets compared to vanilla GPT-4.
  • Severe low-resource language bias: Existing LLM judges perform drastically worse on low-resource languages (e.g., Swahili, Javanese), frequently missing successful jailbreaks.
  • Symmetric utility: A reliable judge doubles as an optimization objective: it can guide gradient/black-box attacks to find subtle exploits (JailBoost) and filter adversarial prompts during deployment (GuardShield).

Key Results

  • Superior Judge Performance: On the complex JAILJUDGE ID test set, JailJudge MultiAgent achieved 0.9127 F1 (vs. 0.8015 for GPT-4-qi2023 and 0.7082 for Llama-Guard-2-8B).
  • Lightweight Model Parity: The open-source JAILJUDGE Guard (7B) achieved 0.8793 F1 on JAILJUDGE ID and 0.9849 F1 on JBB Behaviors, matching or exceeding proprietary models.
  • Downstream Attack & Defense:
  • JailBoost boosted attack performance across multiple baselines by an average of 29.24%.
  • GuardShield slashed the average defense Attack Success Rate (ASR) against four strong attacks (AutoDAN, PAIR, AdvPrompter, AmpleGCG) from 40.46% down to 0.15%.

Limitations and Future Work

  • Multilingual Generalization in the 7B Model: While the multi-agent GPT-4 setup achieved 0.6790 F1 on JAILJUDGE OOD, the smaller JAILJUDGE Guard (7B) attained only 0.4505 F1, reflecting its English-centric pretraining and fine-tuning.
  • Computational Overhead: Multi-agent evaluation requires multiple LLM calls per sample, making it better suited for benchmark labeling or dataset distillation than latency-critical, real-time filtering.
  • Future Directions: Expanding instruction tuning to native multilingual open-source base models, refining real-time inference latency, and incorporating multi-turn contextual jailbreak dynamics.

Summary updated

Abstract

Although significant research efforts have been dedicated to enhancing the safety of large language models (LLMs) by understanding and defending against jailbreak attacks, evaluating the defense capabilities of LLMs against jailbreak attacks also attracts lots of attention. Current evaluation methods lack explainability and do not generalize well to complex scenarios, resulting in incomplete and inaccurate assessments (e.g., direct judgment without reasoning explainability, the F1 score of the GPT-4 judge is only 55% in complex scenarios and bias evaluation on multilingual scenarios, etc.). To address these challenges, we have developed a comprehensive evaluation benchmark, JAILJUDGE, which includes a wide range of risk scenarios with complex malicious prompts (e.g., synthetic, adversarial, in-the-wild, and multi-language scenarios, etc.) along with high-quality human-annotated test datasets. Specifically, the JAILJUDGE dataset comprises training data of JAILJUDGE, with over 35k+ instruction-tune training data with reasoning explainability, and JAILJUDGETEST, a 4.5k+ labeled set of broad risk scenarios and a 6k+ labeled set of multilingual scenarios in ten languages. To provide reasoning explanations (e.g., explaining why an LLM is jailbroken or not) and fine-grained evaluations (jailbroken score from 1 to 10), we propose a multi-agent jailbreak judge framework, JailJudge MultiAgent, making the decision inference process explicit and interpretable to enhance evaluation quality. Using this framework, we construct the instruction-tuning ground truth and then instruction-tune an end-to-end jailbreak judge model, JAILJUDGE Guard, which can also provide reasoning explainability with fine-grained evaluations without API costs. Additionally, we introduce JailBoost, an attacker-agnostic attack enhancer, and GuardShield, a safety moderation defense method, both based on JAILJUDGE Guard. Comprehensive experiments demonstrate the superiority of our JAILJUDGE benchmark and jailbreak judge methods. Our jailbreak judge methods (JailJudge MultiAgent and JAILJUDGE Guard) achieve SOTA performance in closed-source models (e.g., GPT-4) and safety moderation models (e.g., Llama-Guard and ShieldGemma, etc.), across a broad range of complex behaviors (e.g., JAILJUDGE benchmark, etc.) to zero-shot scenarios (e.g., other open data, etc.). Importantly, JailBoost and GuardShield, based on JAILJUDGE Guard, can enhance downstream tasks in jailbreak attacks and defenses under zero-shot settings with significant improvement (e.g., JailBoost can increase the average performance by approximately 29.24%, while GuardShield can reduce the average defense ASR from 40.46% to 0.15%).
BibTeX
@misc{liu2024jailjudge,
  author = {Fan Liu and Yue Feng and Zhao Xu and Lixin Su and Xinyu Ma and Dawei Yin and Hao Liu},
  title = {{JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework}},
  year = {2024},
  eprint = {2410.12855},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.12855},
}

Loading related content...

Figure