Beyond Uniform Criteria: Scenario-Adaptive Multi-Dimensional Jailbreak Evaluation

Lai Jiang, Yuekang Li, Xiaohan Zhang, Youtao Ding, Li Pan

Abstract

Current Large Language Model (LLM) safety evaluations often rely on binary "jailbroken" labels or rigid, uniform multi-dimensional criteria that fail to account for context. We present SceneJailEval, a modular, scenario-adaptive framework that tailors evaluation metrics to the specific risk context of a prompt. By mapping queries to 14 distinct scenarios and dynamically weighting evaluation dimensions, SceneJailEval achieves state-of-the-art performance with an F1 score of 0.917. This framework not only improves detection accuracy but also provides a nuanced quantification of harm, revealing distinct vulnerability profiles across leading LLMs.

Introduction

As LLMs are increasingly deployed in sensitive domains, "jailbreaking"—the use of adversarial prompts to bypass safety guardrails—poses a significant security risk. Existing evaluation methods generally fall into two categories: binary classifiers, which lack nuance regarding the severity of harm, and uniform multi-dimensional frameworks, which apply the same criteria regardless of context. For instance, "Relative Truthfulness" is critical when evaluating a fake chemical recipe but irrelevant for a racial slur. SceneJailEval addresses these limitations by arguing that the "yardstick" for safety must adapt to the scenario, whether it involves medical advice, political incitement, or regional legal compliance.

Methodology: The SceneJailEval Framework

SceneJailEval employs a four-stage modular pipeline designed to provide context-aware safety assessments.

SceneJailEval modular pipeline architecture
  1. Scenario Classifier: An LLM-based agent categorizes the input query into one of 14 predefined risk scenarios, such as "Violent Crime," "IP Infringement," or "Regional Sensitive Issues."
  2. Scenario-Dim Adapter: This core component dynamically selects relevant evaluation dimensions and assigns weights based on expert consensus derived via the Delphi method and Analytic Hierarchy Process (AHP).
  3. Jailbreak Detector: Evaluates the response across six dimensions (e.g., Rejection, Helpfulness, Risk Warning). A safety breach is flagged if any relevant criterion is violated.
  4. Harmfulness Evaluator: For detected jailbreaks, this module quantifies harm using dimensions inspired by cybersecurity frameworks (DREAD/CVSS): Authenticity, Specificity, Severity, and Impact Scope.

Experimental Results and Analysis

The framework was validated using the new SceneJailEval Dataset, comprising 1,308 queries across 14 scenarios, including regional-specific cases. SceneJailEval outperformed existing benchmarks, achieving an F1 score of 0.917—a 6% improvement over previous state-of-the-art methods.

Model Vulnerability Profiles

Testing mainstream LLMs revealed that safety is not uniform; models exhibit specific "weak spots" depending on the scenario.

Model vulnerability profiles across 14 risk scenarios

As shown in the evaluation results, GPT-4o demonstrated higher vulnerability in "Sexual Content" (25% Attack Success Rate), while Llama-3-8b was notably susceptible to "IP Infringement" (18.75% ASR). Furthermore, the framework's ability to handle "Regional Sensitive Issues" highlights that safety is often context-dependent based on local laws and cultural taboos.

Discussion and Conclusion

Ablation studies confirmed that the scenario-adaptive component is the primary driver of performance; removing it led to a significant drop in accuracy. SceneJailEval also proved effective at catching "Refusal + Info" traps—where a model provides a standard refusal followed by harmful content—by utilizing the "Helpfulness" dimension in its detector. While the framework currently relies on expert-defined weights for new scenarios, future work aims to automate this process and extend the evaluation to multi-modal inputs.

Summary updated

Abstract

Accurate jailbreak evaluation is critical for LLM red team testing and jailbreak research. Mainstream methods rely on binary classification (string matching, toxic text classifiers, and LLM-based methods), outputting only "yes/no" labels without quantifying harm severity. Emerged multi-dimensional frameworks (e.g., Security Violation, Relative Truthfulness and Informativeness) use unified evaluation standards across scenarios, leading to scenario-specific mismatches (e.g., "Relative Truthfulness" is irrelevant to "hate speech"), undermining evaluation accuracy. To address these, we propose SceneJailEval, with key contributions: (1) A pioneering scenario-adaptive multi-dimensional framework for jailbreak evaluation, overcoming the critical "one-size-fits-all" limitation of existing multi-dimensional methods, and boasting robust extensibility to seamlessly adapt to customized or emerging scenarios. (2) A novel 14-scenario dataset featuring rich jailbreak variants and regional cases, addressing the long-standing gap in high-quality, comprehensive benchmarks for scenario-adaptive evaluation. (3) SceneJailEval delivers state-of-the-art performance with an F1 score of 0.917 on our full-scenario dataset (+6% over SOTA) and 0.995 on JBB (+3% over SOTA), breaking through the accuracy bottleneck of existing evaluation methods in heterogeneous scenarios and solidifying its superiority.
BibTeX
@inproceedings{jiang2026beyond,
  author = {Lai Jiang and Yuekang Li and Xiaohan Zhang and Youtao Ding and Li Pan},
  title = {{Beyond Uniform Criteria: Scenario-Adaptive Multi-Dimensional Jailbreak Evaluation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i42.40866},
}
Figure