When we align Large Language Models (LLMs) using Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), we almost always make a hidden, foundational assumption: human preferences are transitive. We assume that if a judge prefers response $A$ over response $B$, and response $B$ over response $C$, they must logically prefer $A$ over $C$.
This assumption allows us to map human values to a single, linear scale of scalar rewards, typically using the Bradley-Terry model. Under this framework, the probability that a user prefers response $A$ over response $B$ is simply a function of the difference between their scalar rewards: $P(A \succ B) = \sigma(R(A) - R(B))$.
However, real-world human feedback is messy, multi-dimensional, and fundamentally non-transitive. When we force a non-transitive reality into a transitive mathematical model, our alignment algorithms begin to degrade.
Why Transitivity Fails in the Real World
To understand why transitivity fails, consider how humans evaluate complex linguistic outputs. Language models are judged on multiple competing criteria simultaneously: safety, helpfulness, conciseness, creativity, and tone. This multi-attribute evaluation naturally gives rise to preference cycles, much like the game of Rock-Paper-Scissors.
Imagine three model responses:
- Response A is highly creative but slightly verbose.
- Response B is concise and safe, but dry.
- Response C is highly engaging and concise, but skirts the edge of safety guidelines.
A human evaluator weighing these options might prefer $A$ over $B$ because they prioritize creativity over conciseness. They might then prefer $B$ over $C$ because safety is paramount. But when comparing $C$ directly to $A$, they might prefer $C$ because it delivers engagement without the verbosity of $A$. We are left with a cycle: $A \succ B \succ C \succ A$.
This problem compounds dramatically when aggregating feedback from diverse populations. As noted by Ye et al., different users possess conflicting values and cultural norms. When their individual preferences are pooled together, the collective dataset inevitably exhibits cyclical and contradictory rankings—a phenomenon known in social choice theory as the Condorcet paradox. As Hu et al. demonstrate, even automated LLM judges frequently suffer from these transitivity violations, generating inconsistent evaluation graphs that make stable linear ranking impossible.
The Cost of Forcing a Linear Order
What happens when we train an AI using an optimization framework that assumes transitivity where none exists?
As Tang et al. and Liu et al. argue, forcing non-transitive human feedback into a scalar reward function introduces severe model misspecification. Mathematically, a scalar function cannot represent a cycle; it is impossible to assign numbers such that $R(A) > R(B) > R(C) > R(A)$.
Faced with this contradiction, a standard reward model treats the cyclic preferences as random noise. The optimization process attempts to find a compromise line through the cycle, flattening out the nuanced, context-specific trade-offs that human evaluators actually care about. According to Swamy et al., this misspecification causes reward-based optimization to converge to suboptimal regions of the strategy space, leading to policies that oscillate or fail to satisfy collective human constraints.
Alignment as a Zero-Sum Game
To resolve this, a growing body of research abandons absolute scalar rewards entirely, shifting instead toward game-theoretic formulations. If preferences contain cycles like Rock-Paper-Scissors, the goal should not be to maximize an absolute score, but rather to find a strategy that is unbeatable in pairwise competition against any other strategy. This unbeatable strategy is the Nash Equilibrium.
Standard RLHF (Transitive) Nash Optimization (Non-Transitive)
[Response A] (Score: 9.2) [Response A] ─── defeats ───► [Response B]
│ ▲ │
prefers defeats defeats
▼ │ ▼
[Response B] (Score: 7.4) [Response C] ◄─────────────────────┘
Munos et al. introduced Nash Learning from Human Feedback (NLHF), a framework that models preferences directly as pairwise probabilities rather than decomposing them into scalar reward differences. By framing alignment as a symmetric two-player zero-sum game where a policy plays against a copy of itself, the system seeks the Nash policy. This policy represents the optimal mixture of behaviors that satisfies the maximum proportion of human preferences, naturally navigating cyclic structures without getting stuck in an infinite loop of reward maximization.
Building on this foundation, researchers have focused on making these game-theoretic algorithms computationally practical:
- Algorithmic Convergence: In cyclic environments, standard gradient descent algorithms tend to orbit endlessly around the equilibrium rather than landing on it. To solve this, Wang introduced Magnetic Preference Optimization (MPO), a self-play framework that ensures last-iterate convergence to the true Nash equilibrium. Similarly, Zhang et al. proposed Optimistic Nash Policy Optimization (ONPO), utilizing online mirror descent to accelerate convergence and outperform traditional DPO on complex benchmarks.
- Robustness to Noise: Real-world data contains both structural non-transitivity (valuable signal) and random labeler errors (noise). Xu et al. developed Doubly Robust Preference Optimization (DRPO), which decouples alignment from the rigid rationality assumptions of the Bradley-Terry model. This allows general preference models to absorb human inconsistencies without degrading the underlying policy.
Natively Cyclic Mathematical Frameworks
Outside of pure game theory, researchers are redesigning the underlying kernels and representations used to learn preferences from data.
In standard machine learning, we use dot products or distance metrics to compare items, both of which inherently enforce transitivity. To break this limitation, Chau et al. introduced Generalised Preferential Gaussian Processes (GPGP), which utilize a skew-symmetric kernel. A skew-symmetric function $K(x, y) = -K(y, x)$ can represent arbitrary, continuous, bounded pairwise preference patterns. This mathematical property allows the model to natively capture circular dominance patterns—allowing it to map out exactly where the "Rock-Paper-Scissors" dynamics occur in a dataset without forcing a global linear hierarchy.
By moving away from the restrictive assumption of transitivity, modern alignment frameworks no longer view human contradictions as noise to be filtered out. Instead, they treat non-transitivity as a core feature of human judgment, resulting in models that are fundamentally more robust, nuanced, and aligned with the collective values of a diverse world.