Abstract
Modern Large Language Models (LLMs) and Multimodal LLMs (MLLMs) are typically secured through safety alignment processes like Reinforcement Learning from Human Feedback (RLHF). However, these defenses often rely on recognizing specific, well-defined distributions of harmful prompts. This paper introduces JOOD (Jailbreak via OOD-ifying), a framework that demonstrates how transforming malicious inputs into out-of-distribution (OOD) formats—using simple techniques like word-blending and image-mixing—can bypass the safety guardrails of state-of-the-art models, including GPT-4V and OpenAI o1. By inducing uncertainty in the model's safety filters, JOOD successfully elicits harmful responses where traditional attacks fail.
Introduction
Safety alignment has become a cornerstone of AI development, training models to identify and refuse requests that violate safety policies. While effective against "vanilla" harmful instructions (e.g., "How do I build a bomb?"), these guardrails are often brittle when faced with inputs that deviate from the training distribution. The researchers behind this study identify a critical vulnerability: when a harmful prompt is modified to appear "out-of-distribution," the model's internal safety trigger fails to fire. In these scenarios, the model essentially "plays the fool," providing the requested information because it cannot confidently categorize the ambiguous input as a policy violation.
The JOOD Framework
The authors introduce JOOD, a framework that leverages off-the-shelf data augmentation techniques to obfuscate malicious intent. Unlike complex optimization-based attacks, JOOD relies on simplicity to confuse the model's perception:
- Text-Mixing: Harmful keywords are blended with benign ones to create "coined words." For example, mixing "bomb" with "apple" results in "bombapple." The model is then prompted to explain the components of this new term, leading it to describe explosive precursors.
- Image-Mixing (for MLLMs): Using techniques like "Mixup" or "CutMix," a harmful image (e.g., a grenade) is blended with a benign one (e.g., an apple). This visual noise makes it difficult for safety filters to classify the image as dangerous.
- Typographic Attacks: Malicious instructions are embedded directly into images as text, which MLLMs often process through different pathways than raw text inputs.
- Uncertainty Prompting: Attackers ask the model to describe "each of the objects" in a mixed input. Because the model is uncertain about the overall context but can still recognize individual components, it often proceeds to provide detailed, harmful instructions for the malicious element.
The Role of Uncertainty
A central insight of the paper is the direct correlation between model uncertainty and jailbreak success. Safety guardrails are most effective when the model is highly certain that a prompt is malicious. JOOD strategically pushes the model into a "gray zone" where its maliciousness score is high enough for the generative component to understand the request, but the uncertainty is high enough to prevent the safety filter from triggering a refusal.
Furthermore, the researchers found that semantic distance plays a role: mixing a harmful object with a semantically dissimilar benign object (e.g., a bomb mixed with "softwood") is more effective than mixing it with something similar. The increased "weirdness" of the combination maximizes the model's classification uncertainty.
Experimental Results
The study evaluated JOOD against several benchmarks, including AdvBench, AdvBench-M, and MM-SafetyBench. The results indicate that even the most advanced proprietary models are vulnerable:
- GPT-4V: Achieved a 63% Attack Success Rate (ASR) in explosive-related scenarios, compared to a 0% success rate for standard vanilla attacks.
- OpenAI o1: Despite its advanced reasoning capabilities and robust safety training, o1 was vulnerable to JOOD, which achieved a 52% ASR on the Physical Harm scenario of MM-SafetyBench—a benchmark where previous state-of-the-art attacks largely failed.
- Hacking Scenarios: JOOD improved the ASR by +42% over existing baselines like FigStep.
Qualitative analysis showed that GPT-4V, when presented with OOD-ified inputs, provided detailed chemical formulas for explosives and step-by-step instructions for creating hazardous items, demonstrating a complete failure of the safety alignment in the face of OOD data.
Conclusion and Future Work
The success of JOOD highlights a significant gap in current AI safety methodologies. Alignment is currently too narrow, focusing primarily on clear-cut, in-distribution harmful samples. The authors suggest that to build truly robust models, safety training must incorporate "ambiguous" or "mixed" samples to improve the model's ability to discern intent in OOD scenarios. While certain adaptive defenses like AdaShield can mitigate some of the risks, the fundamental challenge of managing model uncertainty remains an open problem in the field of AI safety.
Summary updated