Abstract
Fine-tuning on task-specific data to boost downstream performance is a crucial step for leveraging Large Language Models (LLMs). However, previous studies have demonstrated that fine-tuning the models on several adversarial samples or even benign data can greatly comprise the model's pre-equipped alignment and safety capabilities. In this work, we propose SEAL, a novel framework to enhance safety in LLM fine-tuning. SEAL learns a data ranker based on the bilevel optimization to up rank the safe and high-quality fine-tuning data and down rank the unsafe or low-quality ones. Models trained with SEAL demonstrate superior quality over multiple baselines, with 8.5% and 9.7% win rate increase compared to random selection respectively on Llama-3-8b-Instruct and Merlinite-7b models.
Introduction
Large Language Models (LLMs) undergo extensive alignment processes, such as RLHF or DPO, to ensure they remain helpful and harmless. However, this alignment is notoriously brittle. Even a few epochs of fine-tuning on task-specific data—including seemingly benign instruction sets—can cause a model to "forget" its safety guardrails, a phenomenon known as safety-breaking.
This creates a significant challenge: how can developers enhance a model's task-specific performance without compromising its safety? SEAL (Safety-Enhanced Aligned LLM Fine-tuning) addresses this by identifying conflicts between fine-tuning objectives and safety objectives, treating data selection as a strategic optimization problem.
The SEAL Framework
The core of SEAL is a Bilevel Optimization (BLO) framework designed to automatically filter and rank fine-tuning data. Instead of treating all training samples equally, SEAL learns a data ranker ($\sigma(\omega)$) that assigns weights to each sample in the fine-tuning dataset.
The framework operates on two levels:
- Upper Level: Minimizes loss on a small, trusted "safe" dataset to ensure the model remains aligned.
- Lower Level: Minimizes loss on the weighted fine-tuning dataset.
To make this practical for large-scale models, the authors introduced a memory-efficient gradient-based algorithm. This variant avoids the need to store auxiliary model parameters, enabling full-parameter fine-tuning of large models without prohibitive memory overhead.
Experimental Evaluation
The authors evaluated SEAL across several benchmarks, including Anthropic HH for helpfulness/harmlessness, SlimOrca/RedOrca for general instruction following, and HEX-PHI for safety.
Key Results
- Performance Gains: SEAL achieved an 8.5% win rate increase over random selection on Llama-3-8B-Instruct and a 9.7% increase on Merlinite-7B.
- Safety Preservation: In safety-critical tests using the HEX-PHI benchmark, SEAL-tuned models maintained safety levels close to original aligned models, significantly outperforming standard Supervised Fine-Tuning (SFT).
- Complementary Effects: When paired with SafeInstr (data augmentation), SEAL reached a 77.28% win rate on safety benchmarks, demonstrating that data selection and augmentation are synergistic.
Discussion and Insights
SEAL shifts the focus from model-level constraints to data-centric safety. A key finding is the transferability of the data selector: a ranker trained on a smaller model (e.g., Phi-3-Mini) can effectively select data for larger models (e.g., Merlinite-7B), making the process computationally efficient. Once a dataset is ranked, the weights can be reused across different models or training runs.
While SEAL requires a high-quality "safe" reference set, it provides a robust mathematical foundation for detecting and mitigating data conflicts that lead to safety degradation during fine-tuning.
Summary updated