Abstract
Aligning large language models (LLMs) with human values and intents critically involves the use of human or AI feedback. While dense feedback annotations are expensive to acquire and integrate, sparse feedback presents a structural design choice between ratings (e.g., score Response A on a scale of 1-7) and rankings (e.g., is Response A better than Response B?). In this work, we analyze the effect of this design choice for the alignment and evaluation of LLMs. We uncover an inconsistency problem wherein the preferences inferred from ratings and rankings significantly disagree 60% for both human and AI annotators. Our subsequent analysis identifies various facets of annotator biases that explain this phenomena, such as human annotators would rate denser responses higher while preferring accuracy during pairwise judgments. To our surprise, we also observe that the choice of feedback protocol also has a significant effect on the evaluation of aligned LLMs. In particular, we find that LLMs that leverage rankings data for alignment (say model X) are preferred over those that leverage ratings data (say model Y), with a rank-based evaluation protocol (is X/Y's response better than reference response?) but not with a rating-based evaluation protocol (score Rank X/Y's response on a scale of 1-7). Our findings thus shed light on critical gaps in methods for evaluating the real-world utility of language models and their strong dependence on the feedback protocol used for alignment.
Introduction
The alignment of Large Language Models (LLMs) with human values typically relies on Reinforcement Learning from Human (or AI) Feedback (RLHF/RLAIF). Due to the high cost of detailed "dense" feedback, practitioners often utilize "sparse" feedback. This introduces a fundamental design choice: Ratings (absolute scores on a scale) versus Rankings (relative pairwise comparisons).
While research has focused extensively on alignment algorithms such as PPO or DPO, the structural impact of the feedback protocol itself has remained largely unexamined. This paper investigates whether these two methods are interchangeable and how they influence the training of reward models and the subsequent evaluation of aligned LLMs.
The Inconsistency Problem
A primary contribution of this work is the identification of a significant "inconsistency problem." Preferences inferred from absolute ratings (e.g., assuming A is preferred to B if its rating is higher) disagree with direct pairwise rankings approximately 60% of the time. This high rate of disagreement is consistent across both human and AI annotators (GPT-3.5-Turbo).
Analysis of annotator behavior reveals distinct biases associated with each protocol. Human annotators tend to give higher absolute ratings to denser, more verbose responses. However, when performing pairwise rankings, they shift their priority toward accuracy and factual correctness. This suggests that the "mental yardstick" used by annotators changes depending on the task format.
Methodology
The researchers constructed a systematic pipeline to compare these protocols:
- Data Collection: An Alpaca-7B model generated multiple responses to 5.2K instructions sourced from Dolly, User-orient, and SuperNI.
- Feedback Acquisition: 71K instances of feedback were collected from GPT-3.5-Turbo, alongside a high-quality subset of 6K instances from human annotators.
- Reward Modeling: Regression reward models were trained on absolute ratings, while Negative Log-Likelihood (NLL) reward models were trained on pairwise rankings.
- Alignment and Evaluation: A "Best-of-n" policy (rejection sampling) was employed to select the best responses. These were then evaluated against a reference model (DaVinci-003) using both rating and ranking protocols.
Protocol-Favoritism in Evaluation
The study uncovers a "protocol-favoritism" effect, where the perceived performance of an aligned model is heavily influenced by the evaluation metric used. A model aligned using ranking data performs exceptionally well when the evaluation is also ranking-based. However, its lead over the base model shrinks significantly when evaluated via ratings.
As illustrated in Figure 5, when using human rankings for evaluation, the gap between a ranking-aligned model and the base SFT model was 11.2%. When using human ratings for the same evaluation, that lead dropped to just 5%. This suggests that current evaluation benchmarks may be overestimating model quality simply because the evaluation format matches the training format.
Conclusion
The choice between ratings and rankings in feedback acquisition is not merely a matter of convenience; it fundamentally shapes the resulting model's behavior and its perceived utility. The 60% inconsistency rate and the protocol-favoritism effect highlight critical gaps in current alignment and evaluation methodologies. Future research should focus on developing more robust evaluation protocols that are less sensitive to the choice of feedback acquisition and exploring the cognitive factors that drive annotator inconsistency.
Summary updated