Introduction
Reinforcement Learning from Human Feedback (RLHF) is the dominant paradigm for aligning powerful AI systems to be safe and helpful. However, RLHF is typically modeled as a problem of learning a single preference ranking from noisy feedback. In reality, true human preferences are complex, conflicting, and represent substantive disagreements stemming from diverse values.
To address this, recent research has turned to Social Choice Theory—the mathematical study of voting and collective decision-making—to evaluate how well AI alignment algorithms aggregate diverse preferences. Through this lens, standard RLHF objectives are equivalent to the Borda count voting rule, while newer algorithms align with the game-theoretic von Neumann winner. Yet, previous theoretical analyses have ignored a crucial component of practical RLHF: KL-regularization. This paper bridges the gap between theoretical social choice and practical AI alignment, proving that regularization is not just a training trick to prevent model drift, but a mathematical necessity for fairness and stability.
The Alignment Problem as Social Choice
When aligning an AI, we are essentially trying to aggregate the messy preferences of thousands of human raters into a single model behavior. How do we pick a "winner" when the voters disagree?
Recent studies have shown that standard RLHF (such as the Bradley-Terry model used in InstructGPT and Llama) functions like a Borda count, where candidates are ranked and their scores are summed. Conversely, newer methods like Nash MD or SPO correspond to the von Neumann winner, a game-theoretic approach that finds a probabilistic distribution of outcomes that no other distribution can beat on average.
In practice, every RLHF algorithm employs KL-divergence regularization to keep the model from drifting too far from its original pre-trained state (the reference policy). Theoretical proofs have historically ignored this regularization. This paper asks a critical question: Does this practical hack help or hurt the fairness and stability of our alignment rules?
Evaluating Alignment Rules
The author treats AI alignment as a problem of probabilistic social choice. Instead of selecting one definitive "best" answer, the model outputs a probability distribution over possible answers. The paper evaluates the Borda count and the von Neumann winner against four classic fairness axioms:
- Pareto Optimality: If everyone prefers option A over option B, the model should also prefer A over B.
- Independence of Irrelevant Alternatives (IIA): Adding a "bad" third option should not change the relative ranking of two "good" options.
- Population Consistency: If two different groups of raters agree on a model, merging the groups should not alter that result.
- Strategy-Proofness: A single rater should not be able to "game the system" by lying about their preferences to secure a better outcome.
Using mathematical proofs derived from the properties of KL-divergence and minimax games, alongside synthetic counterexamples (e.g., a 3-alternative, 64-rater setup), the paper demonstrates how strongly convex KL-regularization stabilizes the alignment process.
Key Insights: Regularization as a Fairness Feature
The findings reveal that KL-regularization does much more than prevent language models from degrading or "hallucinating." It fundamentally alters the social choice properties of the alignment algorithm.
- Approximate Strategy-Proofness: Regularization makes it mathematically difficult for a small group of bad actors to hijack the model's behavior. The author proves that in the regularized von Neumann winner, the maximum gain a single rater can achieve by lying is bounded by $O(1/n)$, where $n$ is the number of raters. Without regularization, a single rater can shift the outcome by a constant amount (up to a 4/15 probability shift), regardless of the total number of raters.
- The Borda Count Remains Flawed: Even with regularization, the standard RLHF objective (Borda count) fails the Independence of Irrelevant Alternatives (IIA) test. This means that simply changing which "bad" examples are included in the training set can unexpectedly shift the model's top-ranked behaviors.
- The von Neumann Winner is the Democratic Choice: Because it corresponds to "maximal lotteries," the von Neumann winner is the only rule that avoids the dictatorship trap (Arrow’s Impossibility Theorem) while satisfying key fairness axioms in a probabilistic setting. Regularization only fortifies these properties.
The paper formalizes these benefits in a "Stability Theorem," showing that the output of a regularized alignment rule is bounded by the size of the perturbation divided by the regularization strength ($\tau$). Experimental verification using a Rock-Paper-Scissors preference matrix confirmed that while an unregularized model could be dramatically swung by a single rater, the regularized model remained remarkably stable.
Limitations and Future Directions
While regularization improves fairness, it introduces a strict dependency on the reference policy. The "Pareto Optimality" achieved is only a regularized version: if the starting reference model has a 0% chance of generating a specific response, the aligned model will also have a 0% chance of generating it, even if every human rater unanimously prefers it.
Additionally, the paper operates within a specific mathematical framework of linear preference relations, which may not capture the full nuance of human psychology. Future research should explore other social choice rules that might interact even more synergistically with KL-regularization, paving the way for alignment algorithms that are both highly stable and unequivocally fair.
Summary updated