Robust Alignment with Noisy Preference Data

457 concepts273 papers2 proposals

When aligning Large Language Models (LLMs) using human feedback, standard frameworks like Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF) operate under a fragile assumption: that the preference labels are clean, consistent, and accurately reflect ground-truth human values.

In practice, large-scale preference datasets are highly corrupted. As detailed by Min-Hsuan Yeh et al. in How Reliable Is Human Feedback For Aligning Large Language Models?, human feedback is plagued by annotator fatigue, subjective contradictions, and structural flaws like forced-choice paradigms where annotators must pick a "winner" even when both responses are equally poor or toxic. This introduces significant label noise—frequently flipping the preference label so that an inferior or harmful response is marked as chosen.

The Mechanics of Preference Vulnerability

To understand why label noise is so destructive, we must look at how preference optimization algorithms calculate updates. Algorithms like DPO optimize a policy by maximizing the implicit reward margin between the chosen response $y_w$ and the rejected response $y_l$.

The vulnerability of this mechanism is analyzed by Alizée Pace et al. in Uncertainty-Penalized Direct Preference Optimization. Standard preference loss functions apply the strongest gradient updates to samples where the model's current predictions deviate most from the provided labels. When a label is flipped due to noise, a well-behaved model will initially assign a low reward to the truly inferior response. However, because the noisy label claims this inferior response was "chosen," the optimization algorithm sees a massive prediction error. It responds by aggressively forcing the policy to adjust, effectively hijacking the model to learn spurious patterns and incorrect behaviors. This phenomenon leads directly to policy degeneracy and reward hacking.

Algorithmic De-biasing and Noise-Robust Losses

One major front in addressing this challenge focuses on modifying the optimization objective itself to make it mathematically blind to or tolerant of corrupted labels.

Instead of relying on heuristics like label smoothing, Sayak Ray Chowdhury et al. introduce a theoretically grounded framework in Provably Robust DPO: Aligning Language Models with Noisy Feedback. They propose an unbiased loss estimator that mathematically incorporates the probability of label flips directly into the objective function. By combining the loss of the observed preference pairs with the loss of their inverse counterparts—weighted by a noise rate parameter—the framework cancels out the mathematical bias inherent in the noisy feedback, allowing models to maintain high performance even with noise rates as high as 40%.

Other approaches modify the underlying divergence measures or regularization parameters to prevent over-fitting to anomalies. For instance, the authors of $\alpha$-DPO: Robust Preference Alignment for Diffusion Models via $\alpha$ Divergence replace the standard Kullback–Leibler (KL) divergence with a mode-seeking $\alpha$-divergence, allowing the model to focus on high-confidence preference trends rather than trying to cover every noisy outlier. Similarly, Junkang Wu et al. develop β-DPO: Direct Preference Optimization with Dynamic β, which dynamically scales the regularization parameter $\beta$ at the batch level based on data quality, effectively down-tuning the model's learning rate when encountering high-noise samples.

Data-Centric Filtering and Label Correction

Rather than altering how the model learns from a noisy dataset, a parallel strategy aims to clean, reweight, or correct the dataset before or during the training loop.

A prominent example of explicit label correction is PerpCorrect, introduced by Keyi Kong et al. in Perplexity-aware Correction for Robust Alignment with Noisy Preferences. This approach leverages the model's own internal language modeling capabilities to compute the Perplexity Difference (PPLDiff) between response pairs. Because a highly corrupted or low-quality response typically yields a distinct perplexity profile compared to a well-structured one, PerpCorrect identifies and flips mismatched labels back to their correct orientation prior to optimization.

[Noisy Preference Dataset] 
       │
       ▼
[Perplexity Difference (PPLDiff) Evaluation] ──► Identifies anomalous "chosen" labels
       │
       ▼
[Label Correction / Flipping] ──────────────────► Restores true preference orientation
       │
       ▼
[Standard Preference Optimization (DPO/RLHF)]

Alternatively, researchers utilize probabilistic frameworks to dynamically weight data points based on estimated reliability. In RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment, the authors treat label correctness as a hidden latent variable, employing an Expectation-Maximization (EM) algorithm to iteratively estimate whether a label is accurate and adaptively down-weighting untrusted pairs. This aligns closely with the work of Rongzhi Zhang et al. in DORM: Preference Data Weights Optimization for Reward Modeling in LLM Alignment, which uses model uncertainty and prediction disagreement to optimize sample weights via bilevel optimization.

This data-centric philosophy is pushed to its logical conclusion by Xun Deng et al. in Less is More: Improving LLM Alignment via Preference Data Selection. Their selection framework, BeeS, demonstrates that filtering out the vast majority of noisy data and training on just the top 10% of high-signal preference pairs yields significantly better alignment outcomes than training on the full, uncleaned dataset.

Decomposing Coarse Binary Preferences

A deeper reason preference data is noisy is that binary sequence-level labels ("A is better than B") compress multi-dimensional human values into a single scalar bit. This compression forces a loss of nuance; for instance, a preferred response might contain a minor factual error, while a rejected response might be completely factual but poorly formatted.

To resolve this, recent methods decompose the preference signal. Zhuocheng Gong et al. propose Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes, which models human preferences as a mixture of multiple discrete latent factors (such as helpfulness versus safety). By mapping feedback to these latent structures, the system can isolate and ignore arbitrary label noise that contradicts the learned sub-dimensions.

Furthermore, moving from binary wins to continuous representations helps mitigate noise. Shiqi Wang et al. develop Reward Difference Optimization For Sample Reweighting In Offline RLHF, which learns a continuous "preference strength" model. By explicitly modeling how much one response is preferred over another, the framework naturally downweights low-confidence, ambiguous comparisons where human annotators are most likely to provide noisy or arbitrary labels.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 275 papers
  2. Version 1 · 2026-10-04 · 472 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_preference_label_noise_bd7727d1,
  title = {preference label noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preference-label-noise/rdr_bd7727d1b5f540efa34b3921304d71f4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_preference_label_noise_bd7727d1_v2,
  title = {preference label noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preference-label-noise/rdr_bd7727d1b5f540efa34b3921304d71f4/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake