Jailbroken: How Does LLM Safety Training Fail?

Alexander Wei, Nika Haghtalab, Jacob Steinhardt

TL;DR

Researchers at UC Berkeley have identified two fundamental failure modes in LLM safety training: competing objectives and mismatched generalization. By exploiting these vulnerabilities, they achieved a near 100% success rate in "jailbreaking" state-of-the-art models like GPT-4 and Claude v1.3. Their findings suggest that safety mechanisms must be as sophisticated as the models they protect to be truly effective.

What Problem Does This Paper Address?

Despite extensive "alignment" via Reinforcement Learning from Human Feedback (RLHF) and red-teaming, Large Language Models (LLMs) remain vulnerable to jailbreaks—adversarial prompts that trick the model into generating harmful content, such as instructions for illegal acts.

Most existing jailbreaks, like the famous "DAN" persona, are discovered through trial-and-error. This paper moves beyond ad-hoc discovery to ask a more fundamental question: Why do these attacks work in the first place? The authors argue that jailbreaks aren't just "bugs" to be patched; they are inherent structural consequences of how we currently train and optimize these models.

Methods: Two Core Failure Modes

The authors hypothesize and test two primary failure modes that allow jailbreaks to succeed:

  1. Competing Objectives: LLMs are trained to be both helpful (follow instructions) and harmless (refuse unsafe requests). An attacker can craft a prompt where these goals conflict. For example, by using Prefix Injection (forcing the model to start its response with "Absolutely! Here is..."), the model’s pretraining objective to complete a coherent sentence overrides its safety objective to refuse.
  2. Mismatched Generalization: Safety training is often conducted on a smaller, narrower dataset than the massive corpus used for pretraining. This creates a gap where the model has a capability (e.g., understanding Base64, ROT13, or obscure languages) that the safety filter hasn't been trained to recognize.
Examples of jailbreak attacks exploiting competing objectives (prefix injection on GPT-4) and mismatched generalization (Base64 encoding on Claude v1.3).

Based on these principles, the researchers designed 30 different attack methods, ranging from simple obfuscation to complex "combination attacks" that layer multiple strategies.

Datasets

The study utilized two primary datasets to evaluate the attacks alongside a harmless control set:

  • Curated Dataset (32 prompts): A high-quality set of harmful requests adapted from OpenAI and Anthropic’s own red-teaming reports (e.g., requests for help with money laundering or synthesizing pathogens).
  • Synthetic Dataset (317 prompts): A larger set of harmful prompts generated by GPT-4 and filtered to ensure they were requests that the models would normally refuse.

Key Insights

  • The "Pretraining Heavy-Lift": When a model is forced into a specific start (like "Absolutely!"), the high-probability tokens from its pretraining (which is mostly helpful, non-refusal data) take over. Once the model starts down the path of answering, it’s much harder for the safety objective to "kick in" and stop the sequence.
  • The Scaling Paradox: Interestingly, scaling models up doesn't necessarily make them safer. In fact, larger models like GPT-4 are more vulnerable to certain attacks (like Base64) than smaller models because they have the sophisticated reasoning required to decode the obfuscation. The smaller model simply fails to understand the input at all.
  • Safety-Capability Parity: This is the paper's core philosophical takeaway. If a model is smart enough to speak a language or follow a complex persona, its safety mechanism must be equally smart. If the safety training doesn't generalize as broadly as the model's capabilities, an attacker will always find a "blind spot."

Key Results

The results were stark, demonstrating that current safety training is far from robust against systematic attacks.

Evaluation results showing the success rates of various jailbreak attacks against GPT-4 and Claude v1.3. Combination and adaptive attacks reach near 100% success.
  • 100% Success Rate: On the curated red-teaming prompts, an "adaptive attacker" (one who tries multiple strategies) achieved a 100% success rate against both GPT-4 and Claude v1.3.
  • Combination Attacks Rule: While simple attacks had moderate success, combining them (e.g., Base64 + Refusal Suppression + Prefix Injection) was devastatingly effective, succeeding on 96% to 99% of all harmful prompts.
  • Claude’s Specific Vulnerability: While Claude v1.3 was very robust against "roleplay" attacks (likely due to targeted training), it remained highly susceptible to "Mismatched Generalization" attacks like Base64.

Limitations and Future Work

The authors acknowledge several limitations, notably the black-box constraints of conducting the study via APIs, meaning they couldn't inspect the internal weights or gradients of the models. Furthermore, this represents a static evaluation; model creators constantly update their models to patch known jailbreaks.

Ultimately, defense is hard. The paper is better at identifying the "why" of the failure than providing a "how" for the fix. The authors suggest that "scaling alone" won't solve the problem and that we may need to move beyond the current "pretrain-then-finetune" paradigm to integrate safety more deeply into the model's core architecture. Future research is encouraged into mechanistic interpretability—looking under the hood to see how these competing objectives are actually represented in the model's neurons.

Summary updated

Abstract

Large language models trained for safety and harmlessness remain susceptible to adversarial misuse, as evidenced by the prevalence of “jailbreak” attacks on early releases of ChatGPT that elicit undesired behavior. Going beyond recognition of the issue, we investigate why such attacks succeed and how they can be created. We hypothesize two failure modes of safety training: competing objectives and mismatched generalization. Competing objectives arise when a model’s capabilities and safety goals conflict, while mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist. We use these failure modes to guide jailbreak design and then evaluate state-of-the-art models, including OpenAI’s GPT-4 and Anthropic’s Claude v1.3, against both existing and newly designed attacks. We find that vulnerabilities persist despite the extensive red-teaming and safety-training efforts behind these models. Notably, new attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests from the models’ red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our analysis emphasizes the need for safety-capability parity—that safety mechanisms should be as sophisticated as the underlying model—and argues against the idea that scaling alone can resolve these safety failure modes.
BibTeX
@inproceedings{wei2023jailbroken,
  author = {Alexander Wei and Nika Haghtalab and Jacob Steinhardt},
  title = {{Jailbroken: How Does LLM Safety Training Fail?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=jA235JGM09},
}
Figure