TL;DR
This paper uncovers a fundamental "feedback inconsistency" in Large Language Model (LLM) alignment: preferences inferred from absolute ratings (e.g., scoring 1-7) disagree with relative rankings (e.g., A vs. B) roughly 60% of the time. Crucially, the authors demonstrate an "evaluation inconsistency," where the perceived superiority of an aligned model depends heavily on whether the evaluation protocol matches the feedback protocol used during its training.
What Problem Does This Paper Address?
Aligning LLMs with human values typically relies on Reinforcement Learning from Human Feedback (RLHF). Researchers generally choose between two "sparse" feedback protocols: ratings (assigning an absolute score to a single response) and rankings (choosing the better of two responses).
While rankings are often preferred for being easier for humans to provide for complex tasks, ratings offer a more granular quantification of quality. The field has largely assumed these protocols are interchangeable signals of the same underlying preference. This paper challenges that assumption, investigating whether the choice of protocol fundamentally changes how models are aligned and, more importantly, how they are evaluated.
Methods
The authors developed a systematic pipeline to compare these protocols:
- Data Collection: They generated multiple responses from a base LLM (Alpaca-7B) for a diverse set of instructions.
- Feedback Acquisition: They collected both ratings and rankings for these responses from human annotators (via Amazon Mechanical Turk) and AI annotators (GPT-3.5-Turbo).
- Consistency Analysis: They converted ratings into a ranking format (e.g., if Response A is rated 5 and B is rated 4, then A > B) and compared them to direct pairwise rankings.
- Reward Modeling: They trained two types of reward models: a regression model for ratings and a Negative Log-Likelihood (NLL) model for rankings.
- Alignment: They used these reward models to implement a "Best-of-n" policy, selecting the highest-scoring response from a pool of candidates.
- Evaluation: Finally, they calculated the win-rates of these aligned models against a reference model (DaVinci-003) using both rating-based and ranking-based evaluation protocols.
Datasets
- Base Model: Alpaca-7B.
- Training Instructions: 5.2K instructions sourced from Dolly, User-orient, and SuperNI.
- AI Feedback: 71K instances from GPT-3.5-Turbo (24.6K ratings, 46.5K rankings).
- Human Feedback: 6K instances (4K ratings, 2K rankings) for agreement analysis.
- Evaluation Set: 553 unseen instructions from OASST, Anthropic, Vicuna, and Koala benchmarks.
- Reference Model: DaVinci-003 (responses from AlpacaEval).
Key Insights and Ideas
- The 60% Disagreement: Preferences inferred from ratings and rankings significantly disagree. For both humans and AI, the consistency score was only about 40-42%.
- Facets of Quality: The inconsistency stems from annotators prioritizing different quality markers depending on the task. For example, qualitative analysis showed that humans might rate a "denser" (more detailed) response higher in isolation, but prefer a more "accurate" response when forced to choose between two in a ranking setup.
- AI Hedging: GPT-3.5-Turbo tends to "hedge" its bets more than humans, frequently labeling pairs as "equal" (57.4% in ratings vs. 40.8% for humans).
- Evaluation Bias: The most striking finding is that evaluation is not objective. A model trained on ranking data looks much better when evaluated via rankings, but its lead shrinks significantly—or disappears—when evaluated via ratings.
Key Results
The authors quantified the "evaluation inconsistency" by measuring win-rates against DaVinci-003. As shown in Figure 3, the choice of protocol creates a self-favoring loop:
- Ranking Evaluation: The Best-of-64 (Rankings) policy showed an 11.2% win-rate improvement over the base SFT model, while the Best-of-64 (Ratings) policy only improved by 5.3%.
- Rating Evaluation: The gap narrowed significantly. The Ratings-based policy improved by 5%, while the Rankings-based policy improved by 4.3%.
This trend held true regardless of whether the evaluator was a human or an AI.
Limitations and Future Work
- Sparse vs. Dense Feedback: The study focuses on sparse feedback (simple scores/choices). The authors suggest that "dense" feedback (detailed textual critiques) might mitigate these inconsistencies but is much more expensive to collect.
- Domain Specificity: The findings are based on open-ended text generation. The dynamics might differ in objective domains like mathematics or code generation, where "correctness" is less subjective.
- Annotator Noise: Human feedback was collected via crowdsourcing, which introduces inherent noise and potential demographic biases.
- Future Directions: The authors call for research into the cognitive reasons why humans provide inconsistent feedback across protocols and the development of more robust evaluation metrics that are less sensitive to the alignment protocol used.
Summary updated