Reward Model Perspectives: Whose Opinions Do Reward Models Reward?

ElleEMNLP 2025

TL;DR

This paper investigates whether reward models (RMs), crucial for aligning language models (LMs) with human values, exhibit sociodemographic biases. They found that RMs, while not perfectly aligned with any single demographic, consistently show relative biases, favoring certain groups over others, and that simple prompting doesn't effectively mitigate these issues.

What Problem Does This Paper Address?

The alignment of large language models (LMs) with human values is a critical area of research. A common approach is Reinforcement Learning from Human Feedback (RLHF), which relies on reward models (RMs) to capture human preferences. However, RMs themselves can inherit biases from their training data, potentially leading to LMs that reflect a narrow set of opinions or even perpetuate harmful stereotypes. This paper addresses the lack of understanding regarding the specific opinions and biases encoded within RMs, asking: whose values are these models actually learning and rewarding?

Methods

The authors propose a framework for measuring the "alignment of opinions" captured by RMs. This involves:

  • Formalizing RM perspectives: They treat RM rewards as indicators of opinions, attitudes, and values.
  • Measuring alignment: They compare the opinion distribution of an RM to that of human respondents (overall and by demographic group) using various "distance" metrics (Jensen-Shannon Distance for non-ordinal data, Wasserstein Distance for ordinal data).
  • Investigating biases: They analyze RMs' performance on datasets designed to test for sociodemographic biases (e.g., OPINIONQA, BBQ, STEREOSET).
  • Testing steerability: They explore whether in-context learning (prompting) can influence RM opinions to better align with specific demographic groups.

Datasets

The study utilizes four existing datasets with sociodemographic labels:

  • BBQ (Parrish et al., 2022): Assesses model biases across age, disability, gender, nationality, race, religion, sexual orientation, and socioeconomic status.
  • OPINIONQA (Santurkar et al., 2023): Elicits opinions on various topics based on personal traits, using data from Pew Research's American Trends Panels.
  • PRISM (Kirk et al., 2024): Contains multi-turn conversations to solicit human feedback for preference alignment, covering demographic features like age, education, and race.
  • STEREOSET (Blodgett et al., 2021): Measures stereotypical biases related to gender, profession, race, and religion.

Key Insights and Ideas

  • Relative vs. Absolute Alignment: The paper distinguishes between absolute alignment (how well an RM matches a specific group's opinions on an absolute scale) and relative alignment (how consistently an RM ranks preferences across different groups). The latter is more critical for preference learning, as it dictates the relative ordering of choices.
  • Consistent Relative Biases: While absolute alignment varies significantly between different RMs, the relative alignment across sociodemographic groups remains remarkably consistent across models. This suggests a systemic issue rather than model-specific quirks.
  • Steering Limitations: Simple in-context learning (prompting) methods like "BIO," "PORTRAY," and "QA" showed little to no statistically significant effect in steering RMs towards better alignment with target demographic groups. This implies that RMs' inherent biases are not easily overcome by such techniques.
  • Stereotype Internalization: RMs can internalize and even reward stereotypes, mirroring biases found in LMs. Some RMs consistently preferred stereotyped choices, while others showed indifference or even a tendency to avoid them.

Key Results

  • Demographic Preferences: RMs consistently align better with certain demographic groups. For instance, on OPINIONQA, RMs tended to align best with individuals from the American South with lower levels of formal education.
  • Stereotype Performance: On the BBQ dataset, BEAVER RM and DEBERTA RM showed a tendency to prefer "Stereotyped" choices, while PYTHIA1B RM and PYTHIA7B RM leaned towards "Unknown" choices.
  • Steering Ineffectiveness: The study found almost no statistically significant effects of steering RMs on OPINIONQA and STEREOSET. In many cases, un-steered models actually outperformed their steered counterparts.
  • RM Consistency: The relative alignment between RMs and demographic groups was found to be consistent across different RMs, with high pairwise Spearman's rank correlations (e.g., 0.67 for OPINIONQA).

Limitations and Future Work

  • Computational Constraints: The authors acknowledge lacking the large-scale compute for more comprehensive experiments, including training RMs and LMs from scratch.
  • Prompt Format Sensitivity: While their experiments suggest robustness to prompt formatting, they only tested one variant of the multiple-choice question format. Future work could explore this further.
  • Dataset Limitations: The analysis is limited to the explored datasets. The authors emphasize the need for more diverse and representative human data with rich sociodemographic metadata.
  • Model Selection: While a comprehensive list of open-source RMs was selected, further exploration with additional models is suggested.
  • Future Research: The authors recommend further study into RM behavior to mitigate unintended consequences, particularly given their critical role in AI alignment and model safety. They also suggest exploring more advanced methods beyond simple prompting for steering RMs.

Summary updated

BibTeX
@inproceedings{elle2025reward,
  author = {Elle},
  title = {{Reward Model Perspectives: Whose Opinions Do Reward Models Reward?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure