SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection

Han Shen, Pin-Yu Chen, Payel Das, Tianyi Chen

Abstract

Fine-tuning on task-specific data to boost downstream performance is a crucial step for leveraging Large Language Models (LLMs). However, previous studies have demonstrated that fine-tuning the models on several adversarial samples or even benign data can greatly comprise the model's pre-equipped alignment and safety capabilities. In this work, we propose SEAL, a novel framework to enhance safety in LLM fine-tuning. SEAL learns a data ranker based on the bilevel optimization to up rank the safe and high-quality fine-tuning data and down rank the unsafe or low-quality ones. Models trained with SEAL demonstrate superior quality over multiple baselines, with 8.5% and 9.7% win rate increase compared to random selection respectively on Llama-3-8b-Instruct and Merlinite-7b models.

Introduction

Large Language Models (LLMs) undergo extensive alignment processes, such as RLHF or DPO, to ensure they remain helpful and harmless. However, this alignment is notoriously brittle. Even a few epochs of fine-tuning on task-specific data—including seemingly benign instruction sets—can cause a model to "forget" its safety guardrails, a phenomenon known as safety-breaking.

Comparison of standard fine-tuning vs. SEAL's weighted approach. Standard fine-tuning (left) can lead to unsafe responses by fitting conflicting data, while SEAL (right) down-ranks unsafe samples to preserve alignment.

This creates a significant challenge: how can developers enhance a model's task-specific performance without compromising its safety? SEAL (Safety-Enhanced Aligned LLM Fine-tuning) addresses this by identifying conflicts between fine-tuning objectives and safety objectives, treating data selection as a strategic optimization problem.

The SEAL Framework

The core of SEAL is a Bilevel Optimization (BLO) framework designed to automatically filter and rank fine-tuning data. Instead of treating all training samples equally, SEAL learns a data ranker ($\sigma(\omega)$) that assigns weights to each sample in the fine-tuning dataset.

The SEAL framework overview. It uses bilevel optimization to train a data ranker that selects high-quality, safe fine-tuning data, resulting in an improved aligned LLM.

The framework operates on two levels:

  1. Upper Level: Minimizes loss on a small, trusted "safe" dataset to ensure the model remains aligned.
  2. Lower Level: Minimizes loss on the weighted fine-tuning dataset.

To make this practical for large-scale models, the authors introduced a memory-efficient gradient-based algorithm. This variant avoids the need to store auxiliary model parameters, enabling full-parameter fine-tuning of large models without prohibitive memory overhead.

Experimental Evaluation

The authors evaluated SEAL across several benchmarks, including Anthropic HH for helpfulness/harmlessness, SlimOrca/RedOrca for general instruction following, and HEX-PHI for safety.

Key Results

  • Performance Gains: SEAL achieved an 8.5% win rate increase over random selection on Llama-3-8B-Instruct and a 9.7% increase on Merlinite-7B.
  • Safety Preservation: In safety-critical tests using the HEX-PHI benchmark, SEAL-tuned models maintained safety levels close to original aligned models, significantly outperforming standard Supervised Fine-Tuning (SFT).
  • Complementary Effects: When paired with SafeInstr (data augmentation), SEAL reached a 77.28% win rate on safety benchmarks, demonstrating that data selection and augmentation are synergistic.

Discussion and Insights

SEAL shifts the focus from model-level constraints to data-centric safety. A key finding is the transferability of the data selector: a ranker trained on a smaller model (e.g., Phi-3-Mini) can effectively select data for larger models (e.g., Merlinite-7B), making the process computationally efficient. Once a dataset is ranked, the weights can be reused across different models or training runs.

While SEAL requires a high-quality "safe" reference set, it provides a robust mathematical foundation for detecting and mitigating data conflicts that lead to safety degradation during fine-tuning.

Summary updated

Abstract

Fine-tuning on task-specific data to boost downstream performance is a crucial step for leveraging Large Language Models (LLMs). However, previous studies have demonstrated that fine-tuning the models on several adversarial samples or even benign data can greatly comprise the model's pre-equipped alignment and safety capabilities. In this work, we propose SEAL, a novel framework to enhance safety in LLM fine-tuning. SEAL learns a data ranker based on the bilevel optimization to up rank the safe and high-quality fine-tuning data and down rank the unsafe or low-quality ones. Models trained with SEAL demonstrate superior quality over multiple baselines, with 8.5% and 9.7% win rate increase compared to random selection respectively on Llama-3-8b-Instruct and Merlinite-7b models. Our code is available on github https://github.com/hanshen95/SEAL.
BibTeX
@inproceedings{shen2025seal,
  author = {Han Shen and Pin-Yu Chen and Payel Das and Tianyi Chen},
  title = {{SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=VHguhvcoM5},
}
Figure