FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization

Mingye Zhu, Yi Liu, Quan Wang, Junbo Guo, Zhendong Mao

Introduction

Large Language Models (LLMs) are typically aligned with human preferences using techniques like Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO). While these methods improve overall helpfulness and safety, they often suffer from a phenomenon called update regression. In these cases, a model loses its ability to correctly handle specific prompts it had previously mastered—a side effect often termed "over-alignment." For instance, a model might become so conservative that it refuses to answer a safe medical query it handled correctly before alignment. These instances of performance backsliding are known as negative flips.

Figure 1: Illustration of update regression. An aligned model may become overly conservative, refusing safe queries that the base model handled correctly. FlipGuard aims to restore this balance.

The FlipGuard Framework

To address update regression, the authors propose FlipGuard, a constrained optimization framework that detects and mitigates negative flips in real-time during the alignment process. The FlipGuard pipeline operates through three primary stages:

1. Reward Characterization

To quantify performance degradation, FlipGuard establishes a reward metric. In the context of RLHF/PPO, this is typically an external Reward Model. For DPO, the framework utilizes an "implicit reward" derived from the log-likelihood of a response under the current policy relative to the reference policy.

2. Detecting Negative Flips

During the training process, FlipGuard continuously monitors the rewards of the pre-aligned model ($\pi_{\theta_0}$) and the active model ($\pi_\theta$). A "negative flip" is identified for a specific data point if the reward of the pre-aligned model exceeds that of the current model by a predefined margin ($\epsilon$).

3. Focal Distillation

Unlike traditional knowledge distillation (KD), which forces the student model to replicate the teacher across the entire dataset, FlipGuard employs focal distillation. It only enforces congruence with the pre-aligned model on the specific samples where regression is detected. This targeted intervention is implemented as an additional Cross-Entropy loss term added to the standard alignment objective, specifically for the regressed samples.

Figure 2: The FlipGuard pipeline. The system identifies negative flips by comparing rewards and applies focal distillation only when regression is detected.

Experimental Evaluation

The effectiveness of FlipGuard was evaluated across four diverse preference datasets: UltraFeedback (general alignment), HH-RLHF (helpfulness and harmlessness), Summarization, and CVALUES (safety and responsibility). Performance was measured using high-capacity models like Llama3-70B and GPT-4 Turbo as judges, alongside standard academic benchmarks such as MMLU, ARC, and GSM8K.

Key Results

The experimental results demonstrate that FlipGuard consistently outperforms standard PPO and DPO baselines:

  • Reduced Negative Flip Rate (NFR): FlipGuard significantly lowers the frequency of regressions. On the UltraFeedback dataset using DPO, the NFR dropped from 55.9% to 46.7%.
  • Preserved Win Rates: Crucially, the reduction in regression does not come at the cost of overall performance. In many instances, FlipGuard actually improved the overall win rate compared to standard alignment methods.
  • Mitigating the Alignment Tax: Alignment often leads to a drop in performance on general knowledge benchmarks. By preventing regression on general tasks, FlipGuard helps preserve the model's intrinsic knowledge, showing superior results on MMLU and ARC.
  • Task-Specific Improvements: Analysis via MT-Bench showed that FlipGuard enhanced model capabilities in Coding, Writing, and Roleplay, though a slight trade-off was observed in complex Reasoning tasks. Robustness analysis also showed that FlipGuard is relatively insensitive to the choice of the hyperparameter $\gamma$ within a reasonable range.

Discussion and Future Directions

FlipGuard demonstrates that alignment does not have to be a zero-sum game between new preferences and existing capabilities. By being "focal" rather than "global," the framework allows for aggressive alignment while maintaining a safety net for previously learned behaviors.

However, some limitations remain. The current implementation relies on KL divergence for the focal constraint; future research could explore alternative distance metrics like Wasserstein distance. Additionally, the slight performance dip in reasoning suggests a need for more nuanced "focal" triggers in complex cognitive scenarios. Extending FlipGuard to a wider array of alignment algorithms also remains a key area for future work.

Summary updated

Abstract

Recent breakthroughs in preference alignment have significantly improved Large Language Models' ability to generate texts that align with human preferences and values. However, current alignment metrics typically emphasize the post-hoc overall improvement, while overlooking a critical aspect: regression, which refers to the backsliding on previously correctly-handled data after updates. This potential pitfall may arise from excessive fine-tuning on already well-aligned data, which subsequently leads to over-alignment and degeneration. To address this challenge, we propose FlipGuard, a constrained optimization approach to detect and mitigate update regression with focal attention. Specifically, FlipGuard identifies performance degradation using a customized reward characterization and strategically enforces a constraint to encourage conditional congruence with the pre-aligned model during training. Comprehensive experiments demonstrate that FlipGuard effectively alleviates update regression while demonstrating excellent overall performance, with the added benefit of knowledge preservation while aligning preferences.
BibTeX
@inproceedings{zhu2024flipguard,
  author = {Mingye Zhu and Yi Liu and Quan Wang and Junbo Guo and Zhendong Mao},
  title = {{FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.960},
}
Figure