Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models

Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, Chun-Ying Huang

Introduction

Large Language Models (LLMs) such as Llama-2 and GPT-4 are typically aligned through Reinforcement Learning from Human Feedback (RLHF) to adhere to strict safety guidelines. However, this alignment is remarkably fragile; recent research demonstrates that even minor fine-tuning—whether on benign or adversarial data—can cause these safety guardrails to collapse, turning a helpful assistant into a potential liability. For users employing Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA (Low-Rank Adaptation), maintaining safety without compromising performance or incurring high computational costs remains a significant challenge.

Methodology: Safe LoRA

The authors propose Safe LoRA, a training-free and data-free approach that treats safety alignment as a geometric projection problem. The core concept is to ensure that weight updates during fine-tuning do not deviate significantly from the model's original safety-aligned subspace.

Overview of the Safe LoRA framework and the projection-based safety guardrail.

The process involves three key steps:

  1. Defining the Alignment Matrix ($V$): The "safety direction" is captured by the difference between a safety-aligned model (e.g., Llama-2-Chat) and its unaligned base version (e.g., Llama-2-Base): $V = W_{aligned} - W_{unaligned}$.
  2. Constructing the Projection Operator ($C$): A projection matrix $C$ is derived from $V$ to map any weight update back onto the safety-aligned subspace. The authors introduced a fast alternative for $C$ that is 250x faster to compute than exact versions.
  3. Selective Projection: During fine-tuning, LoRA updates ($\Delta W$) are monitored. If an update's cosine similarity to its projected version falls below a threshold $\tau$, it is projected back into the safe subspace. Interestingly, only a fraction of layers (e.g., ~11% for Llama-2 and ~35% for Llama-3) typically require this intervention, allowing the model to learn new tasks effectively while keeping guardrails intact.

Experimental Evaluation

Safe LoRA was evaluated on models including Llama-2-7B-Chat, Llama-3-8B-Instruct, and Gemma across several datasets designed to test safety robustness:

  • PureBad: 100 malicious examples collected via red-teaming to simulate direct attacks.
  • Dialog Summary: A mix of 1,000 benign and 100 harmful samples to simulate poisoned real-world data.
  • Alpaca: A standard benign dataset (52k instructions) used to test for unintentional safety degradation.
The eleven safety usage policies used to evaluate model alignment.

The evaluation measured safety across eleven distinct usage policies, as shown in the table above, ranging from illegal activity to tailored financial advice.

Results

Safe LoRA consistently outperformed existing baselines like SafeInstr and Backdoor Enhanced Alignment (BEA) in both safety restoration and utility preservation.

  • Safety Restoration: In attacks using the PureBad dataset, standard LoRA's Attack Success Rate (ASR) skyrocketed to 95.76%. Safe LoRA successfully reduced this to 3.03%, matching the original aligned model's safety level.
  • Utility Preservation: On the MT-Bench utility benchmark, Safe LoRA achieved a score of 6.34, slightly exceeding the original model's score (6.31) and significantly outperforming standard LoRA under attack (4.54).
  • Benign Data Robustness: Even when fine-tuned on benign Alpaca data, where standard LoRA's ASR rose to 86%, Safe LoRA maintained a low ASR of 6.67% while preserving high utility.

Conclusion

Safe LoRA provides a highly efficient, "one-liner" solution for safeguarding LLMs during fine-tuning. By leveraging the geometric properties of safety alignment, it preserves model guardrails without requiring additional safety data or expensive retraining. This makes it a practical and powerful tool for secure model customization in real-world deployments.

Summary updated

BibTeX
@inproceedings{hsu2024safe,
  author = {Chia-Yi Hsu and Yu-Lin Tsai and Chih-Hsun Lin and Pin-Yu Chen and Chia-Mu Yu and Chun-Ying Huang},
  title = {{Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=HcifdQZFZV},
}
Figure