Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs

Yu Yan, Sheng Sun, Zhe Wang, Y. Lin, Z. H. Duan, Zhifei Zheng, Min Liu, Zhiyi Yin, Jianping Zhang

Introduction

As Large Language Models (LLMs) become increasingly integrated into society, ensuring their safety against malicious use is paramount. A primary concern is "jailbreaking"—the process of bypassing a model's safety filters to generate harmful content. Traditionally, the efficacy of these attacks is measured by the Attack Success Rate (ASR). If a model provides a response to a prohibited query, such as "How do I manufacture an explosive?", the attack is deemed successful. However, this metric overlooks a fundamental question: Is the generated content actually dangerous, or is it merely a hallucination wrapped in a toxic persona?

The Problem: The Hallucination Loop

Current evaluation methods often fall into what researchers call a "hallucination loop." In this scenario, a jailbroken model might provide a recipe for a bomb that includes harmless ingredients like salt and sugar. While factually useless, the model delivers this information in a "scary" or "malicious" tone. Automated safety judges, often LLMs themselves, are frequently fooled by this linguistic "vibe," flagging the response as a successful jailbreak despite the lack of any real threat. This creates an illusion of high risk where the model actually lacks the authentic knowledge required to be truly dangerous.

The Hallucination Loop in Jailbreak Evaluation: Distinguishing between authentic knowledge and hallucinated threats.

Methodology: The VENOM Framework

To address these shortcomings, the authors introduce VENOM (Vulnerability Evaluation of Noxious Outputs and Misjudgments). VENOM shifts the focus from mere "breaking" to evaluating the underlying capabilities and the robustness of safety judges through three distinct lenses:

  1. Knowledge-Intensive Q&A: This component tests models on specific, professional-grade criminal knowledge across domains like drug synthesis, explosives, and financial fraud. It utilizes open-ended questions, multiple-choice tests, and fact-verification tasks to gauge the model's actual mastery of harmful information.
  2. Counterfactual Task Planning: This lens evaluates whether a model can plan a crime if it were provided with the necessary knowledge. By "refactoring" harmful tasks into benign ones (e.g., asking for a plan to "eliminate pests" as a proxy for "destroying humanity"), researchers can measure raw logical planning utility without triggering safety filters.
  3. Judge Robustness Testing: This involves deliberately corrupting harmful answers with misinformation while maintaining a malicious tone. This tests whether popular safety judges, such as GPT-4, can distinguish between a genuine threat and a "mean-sounding" hallucination.
Overview of the VENOM Framework: Data collection, task generation, and evaluation pipeline.

Experimental Setup and Datasets

The researchers constructed a comprehensive benchmark for VENOM by aggregating data from high-risk sources, including darknet wikis, chemistry forums, professional manuals for firearms, and financial fraud guides. The resulting dataset comprises over 17,000 raw samples spanning three critical domains: Drug Operations, Explosives & Firearms, and Financial Fraud.

Key Insights and Results

The application of the VENOM framework yielded several critical insights into the current state of LLM safety:

  • The "Malicious Tone" Trap: LLM-as-a-judge frameworks are highly sensitive to linguistic patterns but surprisingly insensitive to factual accuracy. When presented with responses containing 0% factual harmful content but a malicious tone, the False Positive Rate (FPR) reached an alarming 90-100% for some judge frameworks.
  • The Internalization Gap: There is a significant mismatch between a model's susceptibility to jailbreaking and its possession of high-quality criminal knowledge. Most LLMs demonstrate only "fragmentary, outdated, or contaminated" information regarding specialized crimes, achieving reliable mastery in multiple-choice tests only 48% to 58% of the time.
  • Planning vs. Knowledge: While models struggle with specific criminal facts, their planning capabilities remain high. When tasks are refactored into benign contexts, models demonstrate an ability to organize complex, multi-step operations effectively.
  • Open-Source Parity: Interestingly, open-source models like DeepSeek-V3 and Qwen-2.5 show misuse potential comparable to, and sometimes exceeding, closed-source models like GPT-4o in specific high-risk domains.

Discussion and Future Directions

The findings suggest that the primary barrier to real-world misuse of LLMs is currently the models' own "confusion"—their lack of high-quality, actionable criminal data. However, this is a fragile barrier. As models continue to be trained on larger and more diverse datasets, the risk of internalizing dangerous knowledge increases.

Future work must focus on developing "fact-aware" safety judges that can distinguish between stylistic toxicity and functional blueprints for crime. While VENOM currently focuses on three domains, expanding this evaluation to cyberattacks and biological threats is a necessary next step in securing the future of AI.

Summary updated

BibTeX
@misc{yan2025confusion,
  author = {Yu Yan and Sheng Sun and Zhe Wang and Y. Lin and Z. H. Duan and Zhifei Zheng and Min Liu and Zhiyi Yin and Jianping Zhang},
  title = {{Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs}},
  year = {2025},
  eprint = {2508.16347},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.16347},
}
Figure