DORM: Preference Data Weights Optimization for Reward Modeling in LLM Alignment

Rongzhi Zhang, Chenwei Zhang, Xinyang Zhang, Liang Qiu, Haoming Jiang, Yuchen Zhuang, Qingru Zhang, Hyokun Yun, Xian Li, Bing Yin, Tuo Zhao, Chao ZhangEMNLP 2025

TL;DR

This paper introduces DORM, a novel method for optimizing reward models in LLM alignment by dynamically weighting preference data. DORM addresses the challenges of noisy labels and varying sample importance by using a two-stage approach: initial quality estimation based on model uncertainty and prediction disagreement, followed by bilevel optimization to refine these weights. This leads to more data-efficient and robust reward models, ultimately improving LLM alignment.

What Problem Does This Paper Address?

Aligning Large Language Models (LLMs) with human preferences is crucial for ensuring they behave ethically and reliably. A key component of this alignment process is the reward model, which learns to predict human preferences. However, existing methods for training reward models face two significant challenges:

  1. Noisy Preference Labels: Human-annotated preference data, often collected from crowd workers, can be noisy, with studies reporting up to 37% of labels being inaccurate. This noise can degrade the performance of reward models.
  2. Varying Importance of Preference Samples: Not all preference data points contribute equally to learning. Some samples might be highly informative, while others, especially noisy ones, can be detrimental. Current approaches often treat all data points uniformly or rely on expensive, heuristic-based filtering, which can discard valuable information or fail to adapt to the nuances of preference quality.

These limitations highlight a critical need for a method that can automatically assess and leverage the quality of preference data to train more effective reward models.

Methods

DORM tackles the problem of preference data quality through a two-stage approach:

  1. Preference Data Quality Estimation (Initial Weighting):
    • Epistemic Uncertainty: The method estimates epistemic uncertainty using Monte Carlo (MC) dropout. High uncertainty suggests regions where the model is less confident, potentially indicating informative data points.
  2. Prediction Disagreement: It measures disagreement between the model's predictions and the ground-truth labels. High disagreement can signal mislabeled or noisy data.
  3. Prior Weight Calculation: These two measures are combined into a prior weight $w^0 = \exp(u - \gamma q)$, where $u$ is uncertainty and $q$ is disagreement. This formula favors samples with high uncertainty but low disagreement (potentially hard but reliable) and down-weights samples with low uncertainty but high disagreement (potentially noisy or mislabeled). The hyperparameter $\gamma$ controls the balance.
  4. Regularized Bilevel Optimization (Weight Refinement):
    • Bilevel Optimization Framework: DORM formulates the refinement of data weights as a bilevel optimization problem. The lower level trains the reward model using weighted data, while the upper level optimizes the data weights to minimize validation loss.
  5. Prior-based Regularization: The initial prior weights ($w^0$) are incorporated as an L2 regularization term in the upper-level objective. This anchors the weight optimization to the initial quality estimates, stabilizing the process and preventing extreme weight adjustments.
  6. First-Order Hypergradient Method: To address the computational complexity of bilevel optimization, DORM employs a first-order hypergradient approximation method. This avoids expensive second-order Hessian computations, making the approach scalable.

Datasets

The paper evaluates DORM on several publicly available preference datasets:

  • HelpSteer2: A high-quality human-annotated dataset.
  • OpenAssistant2: A human-annotated conversation corpus with multi-dimensional labels.
  • Magpie-QWen-2.5: A synthetic dataset for supervised fine-tuning.
  • OffsetBias: A pairwise preference dataset designed to reduce common biases.
  • WildGuard: A dataset for evaluating LLM safety under various conditions.

For evaluation, DORM uses the RewardBench benchmark for reward model performance and UltraFeedback for policy model evaluation.

Key Insights and Ideas

  • Data Quality is Paramount: The paper strongly emphasizes that the quality of preference data, not just its quantity, is a critical factor for successful LLM alignment.
  • Uncertainty and Disagreement as Quality Signals: DORM demonstrates that a combination of model uncertainty and prediction disagreement can effectively estimate the quality and informativeness of preference data.
  • Dynamic Weighting Outperforms Static Filtering: By dynamically adjusting data weights during training via bilevel optimization, DORM can adapt to evolving model understanding and better leverage informative samples while mitigating the impact of noisy ones.
  • Bilevel Optimization for Data Curation: Framing data weighting as a bilevel optimization problem allows for a principled, data-driven approach to curating training data, leading to more robust models.
  • Efficiency through First-Order Methods: The adoption of a first-order hypergradient method makes bilevel optimization computationally feasible for large-scale LLM training.

Key Results

  • State-of-the-Art Reward Modeling: DORM, trained on only 50k preference samples, achieves 90.5% accuracy on RewardBench, matching or exceeding models trained on significantly larger datasets (e.g., 10-40x more data).
  • Improved Policy Alignment: LLMs fine-tuned using reward models trained with DORM achieve a 61.2% win rate against baseline methods in downstream alignment tasks, demonstrating the effectiveness of DORM's improved reward signals.
  • Robustness to Noise: DORM consistently outperforms standard SFT on noisy preference labels, with performance gaps widening as noise levels increase, highlighting its robustness.
  • Data Efficiency: DORM achieves comparable performance to strong baselines using substantially less training data, showcasing its data efficiency.
  • Component Effectiveness: Both the prior weighting and bilevel optimization components individually contribute to performance gains, and their integration yields the best results, indicating they are complementary.

Limitations and Future Work

  • Dependence on Validation Set Quality: The effectiveness of DORM's bilevel optimization relies on a high-quality validation set. The authors acknowledge that a low-quality validation set could lead to suboptimal weight adjustments. Future work could explore automated validation data curation or self-refinement techniques.
  • Dataset-Level Weighting: The current implementation assigns weights at the dataset level for computational efficiency. The authors suggest that future work could explore hierarchical weight optimization strategies for finer-grained assignments within datasets.
  • Orthogonal Approaches: The authors note that their approach is orthogonal to data curation methods like those used in Skywork, suggesting potential for combining DORM's reweighting mechanism with cleaner datasets or more advanced curation techniques for further improvements.

Summary updated

BibTeX
@inproceedings{zhang2025dorm,
  author = {Rongzhi Zhang and Chenwei Zhang and Xinyang Zhang and Liang Qiu and Haoming Jiang and Yuchen Zhuang and Qingru Zhang and Hyokun Yun and Xian Li and Bing Yin and Tuo Zhao and Chao Zhang},
  title = {{DORM: Preference Data Weights Optimization for Reward Modeling in LLM Alignment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure