TL;DR
Direct Preference Optimization (DPO) is highly sensitive to noisy or imperfect preference data, which can distort training dynamics through a phenomenon called "gradient dominance." The authors introduce CompassDPO, a reward-free framework that stabilizes training by dynamically regulating the direction and magnitude of mini-batch updates using only internal training signals. Experiments demonstrate significant improvements in safety alignment robustness across multiple LLM backbones, especially under label-flip noise, without adding noticeable computational overhead.
What Problem Does This Paper Address?
Safety alignment for Large Language Models (LLMs) often relies on Direct Preference Optimization (DPO) due to its simplicity and efficiency compared to traditional Reinforcement Learning from Human Feedback (RLHF). However, DPO is notoriously fragile when faced with imperfect, ambiguous, or noisy preference labels common in real-world datasets.
The authors identify a key batch-level failure mode called gradient dominance: a small subset of unreliable, conflicting, or high-loss preference pairs can disproportionately hijack the mini-batch gradient energy, distorting the optimization trajectory. Existing robust DPO variants try to fix this globally (e.g., via static margin adjustments) or require expensive external reward models to filter data. This paper addresses the lack of fine-grained, training-time control over how individual samples shape batch-level optimization dynamics.
Methods
CompassDPO introduces a two-pronged, margin-guided dynamics control mechanism that operates entirely during training without requiring external reward models or data relabeling:
- Directional Control: Early in training, the model establishes a preliminary preference direction. After a warm-up period (set to 30% of training), CompassDPO identifies pairs that strongly conflict with this emerging direction (i.e., samples with highly negative implicit margins). It applies a sparse, budgeted loss-mixing strategy, softly blending the standard DPO loss with its reversed-direction counterpart for these high-conflict pairs to neutralize their disruptive impact.
- Magnitude Control: Even after directional correction, extreme high-loss tail samples can dominate the batch. Instead of discarding them entirely—which loses valuable hard-negative signals—CompassDPO uses adaptive soft winsorization. It calculates a batch-dependent threshold and softly caps the excessive portion of tail losses based on the batch's overall margin statistics.
Both controls rely purely on the implicit DPO reward margin already computed during standard training.
Datasets
The authors evaluated their method using a comprehensive suite of safety and utility benchmarks:
- PKU-SafeRLHF-30K: The primary dataset used for training (26.9K pairs) and in-distribution (IID) testing (3.0K pairs).
- Out-of-Distribution (OOD) Safety Benchmarks: Do-Not-Answer, HarmBench, HH-RLHF, and Salad Bench.
- General Capability & Over-refusal Benchmarks: MMLU-Pro, MT-Bench, and OR-Bench (including OR-Hard-1K for over-refusal and OR-Toxic for unsafe inputs).
Key Insights and Ideas
- The Mathematical Root of Gradient Dominance: The paper provides a local gradient analysis proving that DPO's logistic loss inherently amplifies negative-margin pairs. As training progresses, gradient energy concentrates heavily on a minority of bad or noisy samples, causing validation performance to stagnate.
- Soft Winsorization > Hard Truncation: Completely dropping high-loss tail samples (hard truncation) hurts the model's general utility and leads to severe over-refusal behaviors. Softly capping them preserves the useful gradient directions of hard examples while keeping their magnitude in check.
- Self-Contained Robustness: Robustness doesn't require a separate heavy reward model or data relabeling pipeline; the implicit DPO margin itself serves as an effective training-time compass.
Key Results
The authors evaluated CompassDPO across four diverse backbones: Pythia-2.8B, Llama-3.2-3B, Llama-3-8B, and Qwen2.5-7B.
- Superior Safety Alignment: CompassDPO consistently achieved the highest Win Rate (WR) and lowest Attack Success Rate (ASR) compared to vanilla DPO and robust variants like Dr.DPO and rDPO. For instance, on Qwen2.5-7B, it achieved a 90.23% Win Rate and reduced the Nemotron Safety Guard ASR to 0.00%.
- Out-of-Distribution Generalization: CompassDPO achieved the lowest average ASR across external benchmarks, demonstrating that stabilizing training dynamics prevents the model from overfitting to dataset-specific noise patterns.
- Label-Flip Robustness: Under synthetic label noise (where up to 30% of preference labels were randomly flipped), CompassDPO degraded much more gracefully than baselines. For Pythia-2.8B at a 30% flip rate, CompassDPO maintained a 55.14% WR, whereas vanilla DPO dropped to 41.82%.
- Negligible Overhead: Because the operations are lightweight batch-level statistics, the wall-clock training time remained virtually identical to vanilla DPO (e.g., 46m 47s vs 46m 35s for Llama-3-8B on two H100 GPUs).
Limitations and Future Work
- Over-refusal: Like most safety-aligned models, CompassDPO still suffers from high over-refusal rates on benign but challenging prompts (as evaluated on OR-Bench). The authors note that combining their method with datasets that better cover benign but tricky queries is a crucial next step.
- Batch Size Sensitivity: The framework relies on batch-level statistics; extremely small batch sizes might yield less reliable quantile and margin estimates.
- Domain Scope: The evaluation focused heavily on safety alignment and one helpfulness dataset. Future work is needed to validate its performance across broader preference-learning domains and multiple languages.
Summary updated