Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models

Hritik Bansal, John Dang, Aditya Grover

TL;DR

This paper uncovers a fundamental "feedback inconsistency" in Large Language Model (LLM) alignment: preferences inferred from absolute ratings (e.g., scoring 1-7) disagree with relative rankings (e.g., A vs. B) roughly 60% of the time. Crucially, the authors demonstrate an "evaluation inconsistency," where the perceived superiority of an aligned model depends heavily on whether the evaluation protocol matches the feedback protocol used during its training.

What Problem Does This Paper Address?

Aligning LLMs with human values typically relies on Reinforcement Learning from Human Feedback (RLHF). Researchers generally choose between two "sparse" feedback protocols: ratings (assigning an absolute score to a single response) and rankings (choosing the better of two responses).

While rankings are often preferred for being easier for humans to provide for complex tasks, ratings offer a more granular quantification of quality. The field has largely assumed these protocols are interchangeable signals of the same underlying preference. This paper challenges that assumption, investigating whether the choice of protocol fundamentally changes how models are aligned and, more importantly, how they are evaluated.

Methods

The authors developed a systematic pipeline to compare these protocols:

  1. Data Collection: They generated multiple responses from a base LLM (Alpaca-7B) for a diverse set of instructions.
  2. Feedback Acquisition: They collected both ratings and rankings for these responses from human annotators (via Amazon Mechanical Turk) and AI annotators (GPT-3.5-Turbo).
  3. Consistency Analysis: They converted ratings into a ranking format (e.g., if Response A is rated 5 and B is rated 4, then A > B) and compared them to direct pairwise rankings.
  4. Reward Modeling: They trained two types of reward models: a regression model for ratings and a Negative Log-Likelihood (NLL) model for rankings.
  5. Alignment: They used these reward models to implement a "Best-of-n" policy, selecting the highest-scoring response from a pool of candidates.
  6. Evaluation: Finally, they calculated the win-rates of these aligned models against a reference model (DaVinci-003) using both rating-based and ranking-based evaluation protocols.
Figure 1: Overview of our pipeline to study the effect of the choice between sparse feedback pro- tocols (ratings and rankings) for the alignment and evaluation of LLMs. First, we sample multiple responses from the LLM for the queries in the instructions dataset. Then, we acquire rankings and rating feedback data from the human and AI annotators, independently. Subsequently, the feedback data is used to train the reward models for the Best-of-n policy. Finally, we compute the win-rate against the reference model under ratings and rankings feedback protocol from humans and AI.

Datasets

  • Base Model: Alpaca-7B.
  • Training Instructions: 5.2K instructions sourced from Dolly, User-orient, and SuperNI.
  • AI Feedback: 71K instances from GPT-3.5-Turbo (24.6K ratings, 46.5K rankings).
  • Human Feedback: 6K instances (4K ratings, 2K rankings) for agreement analysis.
  • Evaluation Set: 553 unseen instructions from OASST, Anthropic, Vicuna, and Koala benchmarks.
  • Reference Model: DaVinci-003 (responses from AlpacaEval).

Key Insights and Ideas

  • The 60% Disagreement: Preferences inferred from ratings and rankings significantly disagree. For both humans and AI, the consistency score was only about 40-42%.
  • Facets of Quality: The inconsistency stems from annotators prioritizing different quality markers depending on the task. For example, qualitative analysis showed that humans might rate a "denser" (more detailed) response higher in isolation, but prefer a more "accurate" response when forced to choose between two in a ranking setup.
  • AI Hedging: GPT-3.5-Turbo tends to "hedge" its bets more than humans, frequently labeling pairs as "equal" (57.4% in ratings vs. 40.8% for humans).
  • Evaluation Bias: The most striking finding is that evaluation is not objective. A model trained on ranking data looks much better when evaluated via rankings, but its lead shrinks significantly—or disappears—when evaluated via ratings.

Key Results

The authors quantified the "evaluation inconsistency" by measuring win-rates against DaVinci-003. As shown in Figure 3, the choice of protocol creates a self-favoring loop:

  • Ranking Evaluation: The Best-of-64 (Rankings) policy showed an 11.2% win-rate improvement over the base SFT model, while the Best-of-64 (Ratings) policy only improved by 5.3%.
  • Rating Evaluation: The gap narrowed significantly. The Ratings-based policy improved by 5%, while the Rankings-based policy improved by 4.3%.

This trend held true regardless of whether the evaluator was a human or an AI.

Figure 3: Win-rate against DaVinci-003 of Alpaca-7B LLM and Best-of-64 policy that leverages the reward models trained with the ratings and rankings feedback protocols, independently. The gap between base LLM and Best-of-64 (Rankings) is higher than base LLM and Best-of-64 (Ratings) using the rankings feedback protocol for model evaluation. However, the gap between base LLM and Best-of-64 (Ratings) is slightly higher than the gap between base LLM and Best-of-64 (Rankings) using the ratings feedback protocol for model evaluation. The trend holds for humans and GPT-3.5- Turbo as the evaluators. The error bars represent the 95% CI.

Limitations and Future Work

  • Sparse vs. Dense Feedback: The study focuses on sparse feedback (simple scores/choices). The authors suggest that "dense" feedback (detailed textual critiques) might mitigate these inconsistencies but is much more expensive to collect.
  • Domain Specificity: The findings are based on open-ended text generation. The dynamics might differ in objective domains like mathematics or code generation, where "correctness" is less subjective.
  • Annotator Noise: Human feedback was collected via crowdsourcing, which introduces inherent noise and potential demographic biases.
  • Future Directions: The authors call for research into the cognitive reasons why humans provide inconsistent feedback across protocols and the development of more robust evaluation metrics that are less sensitive to the alignment protocol used.

Summary updated

Abstract

Aligning large language models (LLMs) with human values and intents critically involves the use of human or AI feedback. While dense feedback annotations are expensive to acquire and integrate, sparse feedback presents a structural design choice between ratings (e.g., score Response A on a scale of 1-7) and rankings (e.g., is Response A better than Response B?). In this work, we analyze the effect of this design choice for the alignment and evaluation of LLMs. We uncover an inconsistency problem wherein the preferences inferred from ratings and rankings significantly disagree 60% for both human and AI annotators. Our subsequent analysis identifies various facets of annotator biases that explain this phenomena, such as human annotators would rate denser responses higher while preferring accuracy during pairwise judgments. To our surprise, we also observe that the choice of feedback protocol also has a significant effect on the evaluation of aligned LLMs. In particular, we find that LLMs that leverage rankings data for alignment (say model X) are preferred over those that leverage ratings data (say model Y), with a rank-based evaluation protocol (is X/Y's response better than reference response?) but not with a rating-based evaluation protocol (score Rank X/Y's response on a scale of 1-7). Our findings thus shed light on critical gaps in methods for evaluating the real-world utility of language models and their strong dependence on the feedback protocol used for alignment. Our code and data are available at https://github.com/Hritikbansal/sparse_feedback.
BibTeX
@inproceedings{bansal2024peering,
  author = {Hritik Bansal and John Dang and Aditya Grover},
  title = {{Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=dKl6lMwbCy},
}
Figure