FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization

Mingye Zhu, Yi Liu, Quan Wang, Junbo Guo, Zhendong Mao

Introduction: The Challenge of Update Regression

Aligning Large Language Models (LLMs) with human preferences is a critical step in making them helpful and safe. Techniques like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are standard for this purpose. However, these methods often suffer from a hidden cost known as update regression.

Update regression occurs when a model update improves average performance (the "win rate") but causes the model to fail on specific prompts it previously handled correctly. For instance, a base model might correctly provide instructions on taking sleeping pills, but an aligned version might refuse the query, misidentifying it as dangerous medical advice. This "one step forward, two steps back" behavior leads to inconsistency and erodes user trust.

Figure 1: Illustration of update regression and FlipGuard's solution.

The FlipGuard Framework

To combat this, we introduce FlipGuard, a framework that employs constrained optimization with focal attention to prevent backsliding. Unlike traditional methods that apply global constraints, FlipGuard intervenes surgically only when a regression is detected.

Core Mechanism

The FlipGuard workflow consists of several key stages:

  1. Reward Characterization: Performance is measured using explicit reward scores (for PPO) or implicit rewards derived from log-likelihood ratios (for DPO).
  2. Negative Flip Detection: A "negative flip" is identified when the pre-aligned model's reward is significantly higher than the newly aligned model's reward for the same prompt.
  3. Focal Distillation: Instead of a global penalty, FlipGuard applies a KL-divergence constraint specifically on these identified negative flips.
  4. Constrained Optimization: The final objective combines standard alignment loss with a conditional cross-entropy loss triggered by the detection of a regression.
Figure 2: The FlipGuard framework architecture.

Experimental Evaluation

FlipGuard was evaluated using Mistral-7B and ChatGLM3-6B across four diverse preference datasets: UltraFeedback (general instructions), HH-RLHF (helpfulness and harmlessness), Summarization, and CVALUES (safety).

Key Results

FlipGuard consistently outperformed standard PPO and DPO baselines:

  • Reduced Negative Flip Rate (NFR): On UltraFeedback with DPO, FlipGuard reduced the NFR from 55.9% to 46.7% (Reward Model) and from 24.1% to 20.6% (Llama-3 70B evaluation).
  • Improved Win Rates: Despite its defensive nature, FlipGuard increased overall win rates. For example, on HH-RLHF with DPO, the win rate rose from 46.6% to 51.6%.
  • Knowledge Preservation: FlipGuard-aligned models maintained higher scores on academic benchmarks like MMLU and ARC compared to standard alignment methods.

Discussion and Insights

The success of FlipGuard highlights several important aspects of LLM alignment:

  • Alignment is not a Zero-Sum Game: Surgical constraints allow models to improve safety without sacrificing general knowledge (the "alignment tax").
  • Avoiding the "Refusal Trap": By anchoring to the pre-aligned state for safe but sensitive queries, FlipGuard prevents over-alignment where models become overly conservative.
  • DPO Stability: FlipGuard acts as a vital safety net for DPO, which can be particularly fragile and damaging to specific reasoning abilities.

Limitations and Future Work

Future research will explore FlipGuard's effectiveness on newer alignment strategies like KTO or IPO. Additionally, investigating alternative distance metrics beyond KL-divergence, such as Wasserstein distance, may further enhance stability. While robust, the framework's sensitivity to the regression threshold ($\epsilon$) remains an area for optimization depending on reward model noise.

Summary updated

BibTeX
@inproceedings{zhu2024flipguard,
  author = {Mingye Zhu and Yi Liu and Quan Wang and Junbo Guo and Zhendong Mao},
  title = {{FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.960},
}
Figure