$\alpha$-DPO: Robust Preference Alignment for Diffusion Models via $\alpha$ Divergence

TL;DR

$\alpha$-DPO introduces a noise-robust framework for aligning diffusion models with human preferences. By replacing the standard Kullback–Leibler (KL) divergence with the more flexible $\alpha$-divergence, the authors shift the optimization from a "mass-covering" to a "mode-seeking" objective. Coupled with a dynamic $\alpha$ scheduling mechanism, $\alpha$-DPO achieves superior image-text alignment and aesthetic quality, even when training on datasets riddled with mislabeled or subjective preference pairs.

The Problem: DPO's "Glass Jaw"

Fine-tuning diffusion models (like Stable Diffusion) to match human tastes usually relies on Direct Preference Optimization (DPO). DPO is popular because it bypasses the complexity of Reinforcement Learning from Human Feedback (RLHF) by eliminating the need for a separate reward model.

However, DPO has a critical vulnerability: it is incredibly sensitive to noisy data. Real-world preference datasets, such as Pick-a-Pic, are inherently messy. They contain mislabeled pairs due to human error and subjective pairs where two images are both acceptable, but one is arbitrarily chosen as the winner.

Expand

Source, p. 2

The authors identify the mathematical root cause of this sensitivity. Standard DPO is equivalent to minimizing the Forward KL divergence. Forward KL is "mass-covering," meaning it attempts to assign probability mass to every point in the target distribution. If the target distribution contains "noise" (bad labels), the model dutifully tries to cover that noise, leading to degraded image quality and poor alignment.

The Solution: $\alpha$-DPO

To address this, the authors propose $\alpha$-DPO, reformulating the preference alignment objective using $\alpha$-divergence.

1. From Mass-Covering to Mode-Seeking: Unlike Forward KL, $\alpha$-divergence (specifically when $\alpha < 1$) promotes "mode-seeking" behavior. Instead of trying to cover the entire noisy dataset, the model focuses on high-density regions—the "true" preferences—and effectively ignores outliers.

Expand

Source, p. 5

1. Theoretical Equivalence: The paper provides a formal proof showing that standard Diffusion-DPO is merely a special case of minimizing Forward KL, justifying the shift to a more robust divergence measure.
2. Dynamic $\alpha$ Scheduling: Because not all samples are equally noisy, the authors introduce a mechanism that adaptively adjusts the $\alpha$ value for each sample during training. It uses the model’s own internal "confidence" (the log-ratio of the preferred vs. rejected sample) to decide how much noise tolerance to apply. If the model is confused by a pair, it treats it as potentially noisy and adjusts $\alpha$ to limit its influence.

Datasets and Evaluation

The researchers evaluated their method using:

• Pick-a-Pic v2: A massive real-world dataset of over 500,000 user preferences.
• Synthetic Noisy Datasets: They took Pick-a-Pic and intentionally flipped labels at various rates (10%, 20%, 30%, and 40%) to stress-test the models' robustness.
• Evaluation Benchmarks: Performance was quantified across different dimensions of quality using HPSv2 (Human Preference Score), CLIP (text alignment), Pick Score, and ImageReward.

Key Insights

• The "Aha" Moment on Divergence: The most significant insight is connecting the DPO loss function to the geometric properties of Forward KL. Realizing that DPO's failure in noisy environments is a fundamental property of the KL divergence—not just a training quirk—allowed the authors to look toward the broader family of $f$-divergences for a solution.
• Robustness is "Free": $\alpha$-DPO achieves this robustness without requiring a separate reward model or multiple rounds of data cleaning. It handles the noise "in-loop" during the optimization process.
• Implicit Classification: The model’s own intermediate loss values act as a surprisingly accurate "internal confidence signal," which the authors successfully leveraged for the dynamic scheduling of $\alpha$.

Key Results

• Superior Noise Tolerance: In synthetic tests with a 40% label-flip rate, $\alpha$-DPO significantly outperformed standard DPO and other robust baselines (like cDPO or rDPO). While other models' performance tanked as noise increased, $\alpha$-DPO remained remarkably stable.
• Real-World Gains: On the standard (unmodified) Pick-a-Pic v2 dataset, $\alpha$-DPO achieved higher HPSv2 and CLIP scores than all competitors.
• Human Evaluation: In a blind study, humans preferred images generated by $\alpha$-DPO over standard SDXL-DPO by a significant margin, noting better text alignment and richer visual details.
• Winning Rate: Against the base SDXL model, $\alpha$-DPO maintained a winning rate of over 70% on HPSv2 even at high noise levels, whereas other methods dropped toward or below 50% (random chance).

Limitations and Future Work

• Hyperparameter Sensitivity: While $\alpha$ is dynamic, the system still relies on a hyperparameter $\mu$ to scale that dynamic adjustment. Finding the optimal $\mu$ currently requires some manual tuning.
• Integration with Other Methods: The paper briefly explores combining $\alpha$-DPO with Step-wise Preference Optimization (SPO) and sees gains, suggesting that $\alpha$-DPO could be a "plug-and-play" module for many existing alignment frameworks.
• Future Direction: Developing a fully automated, parameter-free version of the dynamic scheduling remains a key next step for the authors.