In preference learning and reinforcement learning from human feedback (RLHF), the standard protocol is the pairwise comparison. As Verma et al. note, humans struggle to provide consistent absolute scores (cardinal feedback) but are highly adept at choosing between two options (ordinal feedback). This observation has led to the dominance of the Bradley-Terry model, which decomposes complex preferences into a series of binary duels.
However, relying exclusively on binary choices creates a significant information bottleneck. As the number of items to rank grows, the number of required comparisons scales poorly, and the resulting models often fail to capture the nuanced relative quality found in larger sets. A growing body of research is now focused on moving beyond these binary constraints toward multi-item ranking and listwise optimization.
The Limits of Binary Feedback
The primary appeal of pairwise comparisons is their simplicity, yet this simplicity comes at a cost of efficiency and scalability. Guo et al. point out that pairwise architectures face severe scalability challenges when ranking multiple candidate responses, often requiring iterative tournament-style sorting or ELO-based computations that are computationally expensive.
Furthermore, binary feedback is low-density. Taranovic et al. argue that restricting feedback to pairwise comparisons limits the rate at which a model can learn, particularly in complex environments like robotics. This sentiment is echoed by Wilde et al., who demonstrate that "scale feedback"—where users indicate the magnitude of their preference—can learn reward functions with significantly fewer queries than binary choices.
Beyond efficiency, binary comparisons are prone to systematic errors. Shi et al. have documented "position bias" in Large Language Models (LLMs) acting as judges, where the model favors an answer simply because of its placement in the prompt. These biases become even more complex and fragile when models are forced to evaluate larger sets of candidates through a sequence of pairwise steps.
Scaling to Multi-Item Rankings
To address these bottlenecks, researchers are developing frameworks that handle simultaneous comparisons of multiple items. Sui et al. extended the classic dueling bandits framework to "multi-dueling bandits," allowing for the evaluation of multiple "arms" at once. This approach better reflects real-world scenarios like search engines or recommendation lists, where users provide feedback on a set of items rather than a single pair.
The transition from pairwise to listwise feedback requires new mathematical foundations. While pairwise models often rely on the Bradley-Terry model, multi-item frameworks frequently turn to the Plackett-Luce model. Chen et al. and Xu et al. highlight that extending preference models to accommodate these higher-order structures is essential for capturing the expertise and reliability of users in crowdsourced ranking tasks.
In the context of LLM alignment, Yin et al. propose Permutative Preference Alignment (PPA). They argue that traditional Direct Preference Optimization (DPO) fails because it treats alignment as a series of independent binary classifications. By using a differentiable version of the Normalized Discounted Cumulative Gain (NDCG) metric, the authors show that models can be optimized to understand the holistic ranking of multiple responses, leading to superior performance on alignment benchmarks.
Improving Information Gain
If the goal is to learn a preference model with the least amount of human effort, we must ask the most informative questions. Mukherjee et al. apply "optimal design" theory to this problem, generalizing experimental design from individual items to lists. Their framework, Dope, selects the specific sets of items that, when ranked by a human, provide the maximum reduction in model uncertainty.
Similarly, Kaufmann et al. introduce ResponseRank, which leverages "preference strength" derived from metadata like annotator agreement or response times. By ranking comparisons within local groups, they achieve the same reward model accuracy as standard methods while using 30% less data.
The Path Forward
The shift toward multi-item ranking is not merely a matter of computational efficiency; it is a necessity for complex decision-making. As Lerman et al. and Nikishina et al. point out, real-world queries often involve comparing three or more entities (e.g., "Which of these four laptops is best for video editing?").
Current systems, such as Prometheus 2 developed by Kim et al., are beginning to bridge this gap by merging models specialized in both direct assessment and pairwise ranking. However, as Wen et al. caution, simply increasing the accuracy of these models on pairwise benchmarks may not lead to better downstream performance. The future of the field lies in developing benchmarks and algorithms that treat preference as a structured, multi-dimensional ranking problem rather than a simple binary choice.