Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework

Kihyun Kim, Jiawei Zhang, Asuman Ozdaglar, Pablo A. Parrilo

TL;DR

This paper introduces a novel framework for aligning AI models that moves beyond the "majority-takes-all" approach of standard RLHF. By leveraging social choice theory, the authors develop an algorithm that ensures model outputs are proportional to the actual distribution of human preferences, making AI systems more representative of minority viewpoints and more robust against strategic manipulation.

What Problem Does This Paper Address?

Current AI alignment techniques, like Reinforcement Learning from Human Feedback (RLHF) and the more recent Nash Learning from Human Feedback (NLHF), have a "tyranny of the majority" problem.

When evaluators have conflicting preferences, RLHF typically tries to find a single scalar reward that satisfies the most people. This often results in a "winner-take-all" policy that ignores minority opinions entirely. Even game-theoretic approaches like NLHF tend to converge on the "Condorcet winner"—the option that beats all others in head-to-head matchups—which still results in a deterministic policy that doesn't reflect the diversity of the underlying population.

Furthermore, these systems are often susceptible to strategic manipulation, where a specific group can disproportionately influence the model by misreporting their preferences.

Methods

The authors propose a new framework grounded in Social Choice Theory. Their approach consists of three main pillars:

  1. Feasible Population Inference: They demonstrate that while you can't perfectly recover the exact distribution of evaluator preferences from pairwise comparisons alone, you can infer a "feasible set" of distributions. They derive mathematical upper bounds ($u_i$) for how much of the population could possibly prefer a specific option $i$.
  2. Axiomatic Design: They introduce two new formal requirements (axioms) for alignment:
  3. $\alpha$-Population-Proportional Alignment (PPA): The policy must assign a probability to an option that is at least somewhat proportional to the share of the population that ranks it first.
  4. $\gamma$-Population-Bounded Manipulability (PBM): This limits the "gain" any single group can achieve by lying about their preferences.
  5. The $F^\beta$ Algorithm: They propose a policy where the probability of an output is proportional to its inferred population bound. They add a "softmax" parameter ($\beta$) that allows developers to tune the model:
  6. At $\beta = 0$, the model is perfectly proportional (maximizing PPA).
  7. As $\beta \to \infty$, the model becomes deterministic, choosing the majority favorite (Condorcet winner).

For large language models (LLMs), they implement this as a two-phase offline algorithm: first, training a "selector model" to estimate the population bounds, and second, fine-tuning the LLM to match the resulting proportional policy.

Datasets

The authors validated their approach across three distinct settings:

  • Tabular Recommendation: Using the MovieLens 1M dataset (1,297 users, 20 movies) to simulate movie recommendations based on real human rankings.
  • Synthetic LLM Task: A color-preference task where 1,000 simulated evaluators had diverse favorite colors.
  • Large-Scale LLM Alignment: Using the Alpaca-GPT4 dataset (52k prompts). They categorized preferences based on "Expertise" (Elementary vs. PhD level) and "Style" (Friendly vs. Unfriendly) to see if the model could maintain a proportional mix of these personas.

Key Insights and Ideas

  • The Impossibility Theorem: The paper proves a fundamental trade-off: no system can be both perfectly proportional (PPA) and always pick the majority winner (Condorcet consistency). You have to choose where you want to sit on that spectrum.
  • Bounds from Pairwise Data: A major "aha" moment is that even though pairwise data (A vs. B) is "lossy," it still provides enough structure to place hard polyhedral constraints on what the true population distribution must look like.
  • Inherent Robustness: By constraining the policy based on these estimated population shares, the model becomes naturally resistant to "ballot stuffing" or strategic misreporting by any single group.

Key Results

  • Better Representation: In the MovieLens task, the proposed method achieved significantly higher PPA levels (proportionality) compared to RLHF and NLHF, which both collapsed into deterministic "majority" policies.
  • Tunable Alignment: The $\beta$ parameter worked as intended. In the LLM experiments, increasing $\beta$ allowed the model to transition smoothly from a diverse, proportional responder to a high-win-rate majority responder.
  • Reduced Manipulation: The PBM metric showed that the proposed framework significantly reduced the ability of a single group to "hijack" the model's output compared to standard RLHF.
  • Scalability: The authors successfully fine-tuned a Qwen2.5-3B model using this framework, proving it can work in high-dimensional, real-world LLM settings.

Limitations and Future Work

  • Computational Overhead: The two-phase training process (learning bounds, then the policy) is more complex and computationally expensive than "one-shot" methods like Direct Preference Optimization (DPO).
  • Evaluation Challenges: Accurately measuring the "proportionality" of an LLM in the wild is difficult because group identities (like "PhD student" or "Elementary student") are often implicit and hard to classify perfectly.
  • Lower-Ranked Preferences: The current PPA axiom focuses on the "top choice" of evaluator groups. Future work could extend this to account for full rankings (e.g., "if my first choice isn't available, what is my second choice?").

Summary updated

Abstract

Conventional preference learning methods often prioritize opinions held more widely when aggregating preferences from multiple evaluators. This may result in policies that are biased in favor of some types of opinions or groups and susceptible to strategic manipulation. To address this issue, we develop a novel preference learning framework capable of aligning aggregate opinions and policies proportionally with the true population distribution of evaluator preferences. Grounded in social choice theory, our approach infers the feasible set of evaluator population distributions directly from pairwise comparison data. Using these estimates, the algorithm constructs a policy that satisfies foundational axioms from social choice theory, namely monotonicity and Pareto efficiency, as well as our newly-introduced axioms of population-proportional alignment and population-bounded manipulability. Moreover, we propose a soft-max relaxation method that smoothly trades off population-proportional alignment with the selection of the Condorcet winner (which beats all other options in pairwise comparisons). Finally, we validate the effectiveness and scalability of our approach through experiments on both tabular recommendation tasks and large language model alignment.
BibTeX
@inproceedings{kim2026beyond,
  author = {Kihyun Kim and Jiawei Zhang and Asuman Ozdaglar and Pablo A. Parrilo},
  title = {{Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=Egmvi2RWnj},
}
Figure