Abstract
Direct Preference Optimization (DPO) aligns large language models by optimizing pairwise preferences and has shown remarkable effectiveness as a simple and scalable alternative to RLHF. However, in practice, preference data are often noisy. Existing robust variants of DPO mainly rely on uniform objective modifications or global reweighting. While partially effective, these methods treat noisy samples as a homogeneous source of uncertainty and fail to distinguish between different noise types, leading to sub-optimal alignment robustness. In this work, we show that robust preference alignment benefits from addressing different noise types with targeted interventions rather than uniform regularization. We propose winsorized Direct Preference Optimization (wDPO), a robust LLM alignment approach with hierarchical winsorization. Specifically, wDPO adopts a reward-free hierarchical intervention strategy that leverages only signals already available during DPO training. It first uses the implicit margin from DPO log-ratio to identify heterogeneous noise patterns without relying on external reward models. For hard noise, wDPO performs a data-level intervention by sparsely correcting strongly inconsistent preference pairs. For ambiguous comparisons, it applies a gradient-level intervention through soft winsorization, capping extreme losses in the high-loss tail to prevent weakly informative samples from dominating gradient updates. Extensive experiments on PKU-SafeRLHF and multiple external safety benchmarks demonstrate that wDPO consistently improves preference alignment quality and robustness over vanilla DPO and strong DPO-family baselines, with particularly pronounced gains under controlled label-flip noise.
Introduction
Direct Preference Optimization (DPO) has emerged as a dominant paradigm for aligning Large Language Models (LLMs) with human intent. By bypassing the complex reward model training and reinforcement learning loops of traditional RLHF, DPO offers a more stable and computationally efficient path to alignment. However, the efficacy of DPO is predicated on the assumption of high-quality, accurately labeled preference pairs. In real-world scenarios, datasets are frequently contaminated with noise, ranging from accidental label flips to ambiguous pairs where responses are indistinguishable.
wDPO (Winsorized Direct Preference Optimization) introduces a novel framework to address these vulnerabilities. By categorizing noise into "hard noise" and "ambiguous comparisons," wDPO applies targeted interventions that prevent outliers from hijacking the learning process, resulting in models that are both safer and more robust.
The Challenge: Gradient Dominance and Noise Pathologies
Standard DPO suffers from a "glass jaw" when encountering noisy data. The authors identify two primary pathologies:
- Hard Noise: Occurs when the "preferred" and "rejected" labels are incorrectly swapped.
- Ambiguous Comparisons: Occurs when two responses are so similar that the preference is essentially arbitrary.
In vanilla DPO, these samples generate disproportionately large loss values. Because neural network optimization naturally prioritizes the reduction of the largest losses, these noisy samples exert an outsized influence on gradient updates—a phenomenon termed gradient dominance. This dominance effectively masks the signal from clean data, leading to unstable training trajectories and degraded alignment performance.
Methodology: Hierarchical Winsorization
wDPO employs a two-stage Hierarchical Winsorization strategy that intervenes at both the data and gradient levels, utilizing only the internal signals (log-ratios) generated during the training process.
Stage I: Margin-aware Soft Label Correction (Data-level)
This stage targets "hard noise." The algorithm monitors the "implicit margin"—the model's current relative preference between two responses. If the model's internal logic strongly and consistently contradicts a provided label, wDPO applies a sparse, budgeted "label flip." This allows the model to trust its emerging understanding of the task over a likely erroneous ground-truth label.
Stage II: Gradient-oriented Winsorization (Gradient-level)
To handle ambiguity, wDPO utilizes "Winsorization," a statistical technique for limiting the influence of extreme values. The method identifies the "high-loss tail" within a training batch—the samples causing the most significant confusion—and caps their gradient contribution. This prevents ambiguous outliers from dominating the update while still allowing the model to learn from the broader dataset.
Critically, wDPO is reward-free. Unlike competing robust methods that require an external "teacher" reward model to sanitize data, wDPO relies entirely on the model's own internal confidence scores.
Experimental Evaluation
The researchers evaluated wDPO across several safety-critical benchmarks using model families such as Pythia-2.8B, Llama-3, and Qwen-2.5:
- PKU-SafeRLHF-30K: Used for primary training and in-distribution testing.
- OOD Generalization: Tested via Do-Not-Answer, HarmBench, HH-RLHF, and Salad Bench.
- Synthetic Stress Tests: Datasets were modified with 10%, 20%, and 30% random label flips to measure robustness under extreme corruption.
To quantify gradient dominance, the authors utilized the Herfindahl–Hirschman Index (HHI), an economic metric for market concentration. Their analysis proved that in standard DPO, a tiny fraction of samples often accounts for over 50% of the total gradient energy.
Key Results and Insights
wDPO consistently outperformed vanilla DPO and robust variants like IPO, cDPO, and Dr.DPO:
- Enhanced Safety: On PKU-SafeRLHF (Pythia-2.8B), wDPO achieved a 59.92% Win Rate, significantly higher than the 44.23% achieved by standard DPO.
- Resilience to Corruption: In the 30% label-flip test, where standard DPO performance collapsed, wDPO maintained a win rate exceeding 55%.
- Robustness to Adversarial Attacks: wDPO-trained models demonstrated superior resistance to "jailbreaking" on HarmBench, often reducing the Attack Success Rate (ASR) to near zero.
Conclusion and Future Work
wDPO demonstrates that noise in preference alignment is not a monolith; flipped labels and ambiguous pairs require distinct interventions. By leveraging internal model signals for hierarchical correction and capping, wDPO provides a scalable, reward-free path to robust LLM alignment.
Future research may focus on optimizing the hyperparameter selection process (such as the flip budget and tail quantile) and extending these hierarchical interventions to general helpfulness and complex reasoning tasks beyond safety alignment.
Summary updated