Data with High and Consistent Preference Difference Are Better for Reward Model

Qi Lin, Hengtong Lu, Caixia Yuan, Xiaojie Wang, Huixing Jiang, Wei ChenAAAI 2025 · OpenReview

TL;DR

This paper introduces "Preference Difference" (PD), a novel metric to filter human preference data for training reward models in LLM alignment. By focusing on data pairs with high and consistent preference differences, they demonstrate improved reward model accuracy and better downstream RLHF performance, even for implicit alignment methods like DPO.

What Problem Does This Paper Address?

Large Language Models (LLMs) are powerful but can generate outputs that don't align with human values (e.g., being unhelpful or harmful). Reinforcement Learning from Human Feedback (RLHF) is a common technique to align LLMs, relying on a reward model trained on human preference data. However, this preference data is often noisy, sparse, and expensive to collect. The paper argues that simply using all available preference data might not be optimal, and the quality of the data, specifically the difference in preferences between chosen and rejected responses, is crucial for training effective reward models. Existing methods often focus on improving the reward model's loss function rather than the data itself.

Methods

The core of the paper's contribution lies in its theoretical analysis and the introduction of the Preference Difference (PD) metric.

  1. Theoretical Analysis: The authors start with the Bradley-Terry model, a standard framework for pairwise comparisons. They derive both the expected loss (what the loss should be if we had perfect knowledge of true preferences) and the empirical loss (what the loss is based on the observed data) for reward model training. By calculating the Mean Squared Error (MSE) between these two loss functions, they theoretically show that data with higher and more consistent preference differences leads to a lower MSE, indicating a better alignment between the empirical and expected loss. This suggests that such data is more valuable for training the reward model.
  2. Preference Difference (PD) Metric: Based on their theoretical findings, they propose the PD metric. For a reward model $R$, the PD for a sample pair $(x, y_1, y_2)$ (where $y_1$ is preferred over $y_2$) is calculated as $|R(x, y_1) - R(x, y_2)|$. This metric quantifies the magnitude of the reward model's predicted difference between the two responses.
  3. Data Filtering: The proposed method involves:
  4. Training an initial reward model on the raw preference data.
  5. Using this initial model to calculate the PD for all preference pairs.
  6. Sorting the preference pairs by their PD values from high to low.
  7. Filtering the dataset by keeping only the top-k pairs based on PD (e.g., keeping the top 40% or 60% of data).
  8. Extension to DPO: The authors also show that the PD concept can be extended to implicit reward alignment methods like Direct Preference Optimization (DPO), by defining a similar metric based on the DPO model's probabilities.

Datasets

  • HH-RLHF dataset: This dataset, collected from human feedback, was used for training and evaluating reward models and RLHF alignment. It contains dialogue histories with two subsequent responses, ranked by human annotators for helpfulness and harmlessness. The authors used a subset for training and a separate test set.
  • Alpaca dataset: Used for instruction fine-tuning (SFT phase) of the base models.

Key Insights and Ideas

  • Data Quality Over Quantity: The paper strongly suggests that the quality of preference data, specifically the magnitude and consistency of preference differences, is more important than sheer volume for training effective reward models.
  • Bridging the Loss Gap: The theoretical analysis provides a clear link between data characteristics (high/consistent difference) and the reduction of the gap between empirical and expected loss, which is a fundamental goal in statistical learning.
  • PD as a Proxy for Data Value: The PD metric serves as an effective proxy for identifying preference pairs that are most informative for training a reward model. High PD indicates a strong, clear preference, which is less ambiguous and thus more valuable.
  • Universality of PD: The effectiveness of PD filtering is demonstrated not only for traditional RLHF but also for implicit alignment methods like DPO, suggesting its broad applicability.
  • Bias Mitigation: While not the primary focus, the paper hints that PD filtering might help maintain or even improve alignment performance without significantly exacerbating model biases.

Key Results

  • Improved Reward Model Accuracy: Reward models trained on data filtered by PD achieved higher calibrated accuracy compared to models trained on randomly sampled data or the full dataset, especially as the data proportion decreased.
  • Enhanced RLHF Performance: The RLHF alignment performance (measured by GPT-4 win rates) was consistently better when using PD-filtered data for reward model training across three different LLMs (open_llama_3b, Llama-2-7b, Mistral-7b).
  • Optimal Filtering Ratio: For the HH-RLHF dataset, filtering to keep between 50% and 60% of the data with the highest PD yielded the best results, outperforming the full dataset in many cases.
  • DPO Effectiveness: PD filtering also improved the win rate for DPO models, demonstrating its efficacy for implicit alignment. For Llama-2-7B, PD filtering with 60% of data achieved a 73.63% overall win rate, compared to 66.82% for the full dataset.

Limitations and Future Work

  • Computational Cost: While PD filtering improves data utilization, it still requires an initial reward model training step to calculate PD values.
  • Optimal Filtering Threshold: The optimal filtering ratio (e.g., 50-60%) was found empirically for the HH-RLHF dataset and might vary for different datasets or tasks.
  • Bias Analysis: The paper briefly touches upon PD's influence on model biases, showing that it doesn't significantly worsen them, but a more in-depth analysis could be beneficial.
  • Future Work: The authors suggest further exploring PD's impact on AI risk and bias, and potentially developing more efficient methods for data selection or generation that inherently produce high-PD data.

Summary updated

Abstract

Reinforcement Learning from Human Feedback (RLHF) is a commonly used alignment method for Large Language Models (LLMs). This method relies on a reward model trained on a preference dataset to provide scalar rewards. However, the human-annotated preference data is often sparse, noisy, and costly to obtain, necessitating more efficient utilization. This paper proposes a new metric for better preference data utilization from both theoretical and empirical perspectives. Starting with the Bradley-Terry model, we compute the Mean Square Error (MSE) between the expected loss and empirical loss of the reward model. Our findings reveal that data with higher and more consistent difference result in lower MSE. We therefore propose the Preference Difference (PD), the reward difference between two samples, as a filter for preference data. Experimental results on three open-source models show that reward models trained by filtered data with PD achieve higher calibrated accuracy, as well as better RLHF alignment performance. The conclusion remains consistent when we extend the experiments and theoretical derivations to implicit reward alignment algorithms, such as Direct Preference Optimization (DPO).
BibTeX
@inproceedings{lin2025data,
  author = {Qi Lin and Hengtong Lu and Caixia Yuan and Xiaojie Wang and Huixing Jiang and Wei Chen},
  title = {{Data with High and Consistent Preference Difference Are Better for Reward Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=mFJc4azfhx},
}
Figure