Most modern Large Language Model (LLM) alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rest on a hidden assumption: that human preferences are transitive. This assumption, formalized by the Bradley-Terry (BT) model, posits that if a human prefers response A over B, and B over C, they must necessarily prefer A over C.
However, as Yifan Zhang and Quanquan Gu argue in Beyond Bradley-Terry Models, real-world human judgment is often non-transitive, cyclic, and context-dependent. When we force these "messy" preferences into a single scalar reward value, we lose the nuances of human values, leading to models that fail to capture the multi-faceted nature of helpfulness and safety.
The Failure of Scalar Rewards
The core problem with the Bradley-Terry model is that it treats quality as a one-dimensional ladder. In this view, every response has a "true" score, and the probability of one response beating another is simply a function of the distance between their scores.
But human preferences often look more like a game of Rock-Paper-Scissors. A user might prefer a "concise" answer over a "detailed" one for a quick fact-check, but prefer a "detailed" answer over a "safe" one for a creative task, while still preferring "safe" over "concise" in general. These cycles—where A > B > C > A—cannot be represented by a single number. As Yuanhao Wang notes in Is RLHF More Difficult than Standard RL?, these cycles make it impossible to define a stable scalar utility function.
Alignment as a Two-Player Game
To handle these cycles, a new wave of research reframes alignment as a two-player zero-sum game. Instead of asking "What is the score of this response?", we ask "Which policy would win in a head-to-head comparison?"
In this framework, the goal is not to maximize a reward, but to find a Nash Equilibrium—a policy that is so robust that no other policy can consistently beat it in a preference duel. Remi Munos and colleagues pioneered this with Nash Learning from Human Feedback (NLHF), proving that seeking a Nash equilibrium of a preference model provides better alignment with diverse human values than traditional reward modeling.
From Theory to Algorithms
Moving from game theory to practical training requires new optimization objectives. Several researchers have recently proposed "direct" methods that avoid the need for an explicit reward model:
- Self-Play Preference Optimization (SPPO): Yue Wu and Yiming Yang introduced SPPO, which uses a self-play mechanism to iteratively move the model toward a Nash equilibrium. By treating the model as both the player and the opponent, it can identify and resolve its own internal contradictions.
- Direct Nash Optimization (DNO): Researchers like Ching-An Cheng developed DNO, a post-training algorithm that allows models to self-improve by optimizing against a general preference model, effectively "winning" against previous versions of itself.
- Magnetic Preference Optimization (MPO): One challenge with game-theoretic optimization is stability. Mingzhi Wang introduced MPO to ensure last-iterate convergence. While many game-based methods might circle around the optimal solution, MPO uses a "magnetic" force to pull the model toward the exact Nash equilibrium.
Capturing Multi-Dimensional Values
If we can't use a single scalar, how do we represent the "strength" of a preference? Yifan Zhang proposes the General Preference embedding Model (GPM), which embeds responses into a multi-dimensional latent space. In this space, the preference between two responses is determined by their relative positions, allowing the model to capture complex, non-linear relationships that a one-dimensional scale would miss.
Similarly, Benjamin Heymann suggests using social choice theory in Adaptive Preference Aggregation (APA). By viewing alignment as a collective choice problem among diverse human viewpoints, APA uses a "maximal lottery" to find an equilibrium that reflects the aggregate will of a population, even when individuals disagree or hold cyclic views.
Why It Works: Intuition
The intuition behind this shift is that robustness is a better proxy for alignment than score-maximization.
In a transitive world, the "best" model is the one with the highest score. In a non-transitive world, the "best" model is the one that is unbeatable. By framing alignment as a game, we force the model to defend its responses against all possible alternatives. This process naturally prunes away "exploitable" behaviors—like being overly verbose to trick a reward model—because those behaviors would lose to a more concise, accurate opponent in a preference duel.
As Chenlu Ye and Hanze Dong demonstrate in their work on Online Iterative RLHF, this minimax framing leads to significantly better performance on complex reasoning tasks where "correctness" is multi-faceted and difficult to reduce to a simple reward.