When aligning large language models with human preferences, the standard currency of data is the pairwise comparison: does a user prefer response $A$ or response $B$? Algorithms like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) ingest millions of these binary choices to construct a single, scalar reward function.
However, a growing body of research reveals a fundamental mathematical bottleneck: pairwise comparisons suffer from an inherent identifiability failure. Binary choices are structurally incapable of distinguishing between fundamentally different latent distributions of human preferences. No matter how much data you collect, a model trained purely on pairs cannot tell the difference between a population that is universally indifferent and one that is deeply polarized, nor can it map how preferences for different items correlate with one another.
To overcome this barrier, a new paradigm is emerging that shifts data collection from pairs to ternary (or higher-order) subset comparisons. By introducing a third option, we unlock the geometric constraints necessary to uniquely identify the true, pluralistic structure of human choice.
The Mathematical Blindspot of Binary Choice
To understand why pairwise data fails, imagine a population split exactly down the middle: 50% of users passionately prefer conservative viewpoints, while the other 50% passionately prefer liberal viewpoints. If you present this population with a choice between a conservative response ($A$) and a liberal response ($B$), the aggregate data will show a perfect 50/50 split.
Now imagine a completely different population where every single person is entirely indifferent to politics and flips a coin to choose between $A$ and $B$. The aggregate data is identical: a 50/50 split.
As proven by Lin et al., this is a structural posterior identifiability failure. When we aggregate pairwise preferences without explicit subgroup labels, the distinct preference vectors of underlying subgroups collapse into a single, misleading "average." A reward model trained on this data will optimize for the average, producing a neutral, bland policy that may actively dissatisfy every individual subgroup. Crucially, the authors demonstrate that this is not a statistical sampling issue; increasing the volume of aggregate data does nothing to resolve the fundamental lack of information regarding which specific users hold which preferences.
This non-identifiability poses a severe threat to personalization. Chidambaram et al. formalize this in the context of unobserved preference heterogeneity, showing that binary preference comparisons alone cannot reliably distinguish between different underlying latent preference distributions, making true pluralistic alignment mathematically impossible from pairs alone.
The Geometry of Correlation
The failure of pairwise data extends beyond user disagreement to the relationships between the items themselves. Standard preference models assume that the utility of an item is independent. In reality, human preferences are highly correlated; if you love a highly technical, code-heavy response, you are likely to appreciate similar technical responses, while someone else might find them unreadable.
Why can't pairwise comparisons capture these correlations? Consider three items: $X$, $Y$, and $Z$. Suppose $X$ and $Y$ are virtually identical (e.g., two slight variations of the same Python script), while $Z$ is an entirely different approach (e.g., a conceptual explanation).
If we only observe pairwise choices ($X$ vs $Y$, $X$ vs $Z$, $Y$ vs $Z$), we can determine their mean utilities, but we cannot see how they vary together. As Cherapanamjeri et al. demonstrate, binary choices cannot distinguish between multiple distinct covariance structures in a utility space.
Pairwise View (Blind to Correlation):
[X] <---> [Y] (Is X redundant with Y, or independent?)
[X] <---> [Z]
[Y] <---> [Z]
Ternary View (Triangulating the Space):
[Z]
/ \
/ \
[X] === [Y] (The interaction with Z reveals X and Y are highly correlated)
In a bandit setting, this creates a massive inefficiency. Saha et al. point out that pairwise comparisons cannot distinguish between items that are perfectly correlated and those that are independent with similar mean utilities. Because the algorithm cannot detect that $X$ and $Y$ are structurally redundant, it wastes valuable samples evaluating both, rather than pruning the redundancy and focusing on genuinely distinct options.
How Ternary Comparisons Restore Identifiability
The solution to this fundamental limitation is deceptively simple: add a third item. Moving from pairs to subset comparisons of size $k \ge 3$ (such as "best-of-three" or triplet rankings) fundamentally changes the geometry of the problem.
When a user is presented with three items ($A$, $B$, and $C$), their choice provides a point of triangulation. By observing how the presence of item $C$ alters the relative preference between $A$ and $B$, the mathematical model gains access to the "cross-terms" of human decision-making.
1. Uncovering Latent Subgroups
In the case of unobserved population heterogeneity, a third option breaks the symmetry that masks polarization. If a polarized population chooses among three options, the distribution of the "best" item shifts in ways that a single coin-flip population never could. Chidambaram et al. exploit this mechanism to develop EM-DPO, an algorithm that uses ternary comparisons to successfully cluster unobserved annotator groups and train specialized, pluralistic language models.
2. Reconstructing Covariance Matrices
For learning item correlations, triplet-based rankings provide the exact geometric constraints required to map a shared utility space. Cherapanamjeri et al. prove that moving to three-way rankings provides the necessary information to uniquely identify the covariance matrix of a correlated probit model. This allows practitioners to reconstruct complex human choice patterns and dependencies that remain completely invisible to binary models.
3. Accelerating Learning Rates
In optimization, knowing item correlations allows algorithms to navigate the preference space along a low-rank manifold. Saha et al. demonstrate that by utilizing subset comparisons of size three or larger, an algorithm can identify structural redundancies. This collapses the sample complexity of preference bandits, allowing the learning rate to scale with the intrinsic rank of the correlation matrix ($r$) rather than the total number of items ($n$).
Conclusion
The machine learning community has long treated pairwise comparisons as the gold standard for preference aggregation due to their simplicity. However, the mathematical reality is that pairs discard the vital context needed to understand a diverse, interconnected world. By embracing ternary and higher-order comparisons, modern alignment frameworks can move past the fiction of the "average user" and begin to truly map the rich, correlated, and pluralistic landscape of human values.
Key Foundational Work
Position: Aggregate Preference Optimization Hides a Posterior Identifiability Failure for Pluralistic Alignment (2026) — contributed 7 concept elements to this direction.
Learning Correlated Reward Models: Statistical Barriers and Opportunities (2026) — contributed 4 concept elements to this direction.
Exploiting Correlation to Achieve Faster Learning Rates in Low-Rank Preference Bandits (2022) — contributed 3 concept elements to this direction.