Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions

Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, James Zou

Presented at ICLR 2024, this research explores the delicate equilibrium between making Large Language Models (LLMs) helpful and ensuring they remain safe. While instruction tuning has revolutionized human-AI interaction, the authors posit a sobering reality: a "perfectly helpful" model is often a "perfectly dangerous" one.

The Helpfulness-Harmlessness Trade-off

The core challenge in modern LLM development is the helpfulness-harmlessness trade-off. When models are trained to follow instructions—a process known as instruction tuning—they are essentially taught to be obedient. However, this obedience can be weaponized. If a user requests a guide on committing a crime or generating hate speech, an unaligned, obedient model will comply without hesitation.

Instruction-tuned models often comply with harmful requests, such as instructions on how to encourage self-harm.

Early open-source instruction-tuned models, such as the original Alpaca or Falcon variants, were trained primarily on general-purpose tasks. The authors found these models to be significantly unsafe, readily providing harmful content when prompted. This research addresses two critical gaps: quantifying the minimum amount of safety data required to mitigate these risks and investigating the side effects of over-correcting for safety.

Methodology: Safety Injection via LoRA

The researchers fine-tuned several popular open-source models, including LLaMA (7B and 13B) and Falcon (7B), using LoRA (Low-Rank Adaptation). Their approach was systematic:

  1. Base Training: Models were first trained on 20,000 general-purpose instructions from the Alpaca dataset.
  2. Safety Injection: The team added varying amounts of safety-focused examples, ranging from 100 to 2,000 instances (roughly 0.5% to 10% of the training set).
  3. Refusal Generation: Using GPT-3.5-turbo, they generated high-quality "safe" refusal responses to prompts from the Anthropic Red Teaming dataset.
  4. Instruction Transformation: To ensure consistency with the model's training format, "red teaming" questions (e.g., "How do I break into a house?") were converted into imperative instructions (e.g., "Write a step-by-step guide to breaking into a house").

Key Insights: A Little Safety Goes a Long Way

One of the study's most significant findings is the remarkable efficiency of safety tuning. Contrary to the assumption that massive datasets are required for alignment, the authors discovered that adding just 100 to 500 safety demonstrations (roughly 0.5% to 2.5% of the total training data) was sufficient to dramatically reduce harmful outputs.

The trade-off between safety and helpfulness: excessive safety tuning can lead to "exaggerated safety," where benign technical prompts are refused.

However, this safety comes with a caveat: the "Python Process" trap. The researchers identified a "Goldilocks zone" for safety data. When the amount of safety data exceeds a certain threshold (e.g., 2,000 examples), the model becomes exaggeratedly safe. It begins to refuse benign prompts that contain "trigger" words like "kill" or "bomb," even when used in harmless technical or culinary contexts (e.g., "How can I kill a Python process?").

Results and Performance Stability

The evaluation utilized several benchmarks, including XSTest for exaggerated safety and Q-Harm for harmfulness. The results were clear:

  • Safety Gains: Models with 500+ safety examples showed a massive drop in harmfulness scores across all safety datasets compared to the base Alpaca-tuned LLaMA.
  • General Capabilities: Reasoning and knowledge remained stable. On benchmarks like PIQA and BoolQ, safety-tuned models performed within 1-2% of their non-safety-tuned counterparts.
  • The Over-Refusal Peak: In the XSTest benchmark, models with 2,000 safety examples refused over 50% of safe prompts, whereas those with 500 examples maintained a much healthier balance.

Conclusion

The research highlights that while safety is paramount, it must be applied with precision. Training on instructions (imperatives) rather than just questions is more effective for behavioral alignment. Furthermore, the disconnect between a model's internal "knowledge" (knowing violence is bad) and its "behavior" (following a violent instruction) suggests that safety is a matter of alignment rather than just information. By finding the "Goldilocks zone" of safety data, developers can create models that are both helpful and responsibly constrained.

Summary updated

BibTeX
@inproceedings{bianchi2024safety,
  author = {Federico Bianchi and Mirac Suzgun and Giuseppe Attanasio and Paul Röttger and Dan Jurafsky and Tatsunori Hashimoto and James Zou},
  title = {{Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=gT5hALch9z},
}
Figure