Introduction
Aligning Large Language Models (LLMs) with human values typically involves Reinforcement Learning from Human Feedback (RLHF), which aggregates diverse preferences into a single reward model. This "winner-takes-all" approach tends to favor majority opinions, potentially marginalizing minority viewpoints. In pluralistic societies, AI systems should ideally reflect the distribution of human preferences proportionally. This paper introduces a framework for Population-Proportional Alignment (PPA), ensuring that AI behavior represents the diversity of human perspectives in proportion to their prevalence in the population.
Axiomatic Foundations
The authors bridge social choice theory and machine learning by proposing four fundamental axioms for preference learning:
- Monotonicity: An increase in an option's rank should not decrease its selection probability.
- Pareto Efficiency: If all evaluators prefer option A over B, the policy should favor A.
- Population-Proportional Alignment (PPA): The policy should be at least weakly proportional to the size of the group holding a specific preference.
- Population-Bounded Manipulability (PBM): This limits the influence any single group can exert through strategic misreporting of preferences.
As shown in the table below, existing methods like RLHF (Maximal Borda) and Nash Learning (NLHF/Maximal Lotteries) fail to satisfy all these criteria, particularly PPA and PBM.
Methodology
The proposed framework operates by inferring the feasible set of population distributions directly from pairwise comparison data. By deriving upper bounds for the population share of each preference, the authors construct a policy proportional to these bounds.
A central feature of the framework is the introduction of a softmax relaxation parameter, $\beta$. This parameter allows for a controllable trade-off between majority-rule (high $\beta$) and proportional representation (low $\beta$). This is necessary due to a fundamental "impossibility result" proven in the paper: no system can simultaneously satisfy perfect population proportionality and Condorcet consistency (always selecting the majority favorite).
Empirical Evaluation
The framework was validated across three distinct settings:
- MovieLens 1M: Evaluator rankings over movies to test fundamental theory.
- Synthetic Color Preferences: Fine-tuning Qwen2.5-3B-Instruct to reflect diverse color preferences.
- Alpaca-GPT4: A large-scale dataset where evaluator groups were simulated based on "Expertise" (Elementary vs. PhD) and "Style" (Friendly vs. Unfriendly).
Results and Discussion
Experimental results confirm that the framework effectively navigates the trade-off between performance (Win Rate) and proportionality (PPA).
In the MovieLens task, increasing $\beta$ leads to a Win Rate comparable to RLHF/NLHF (~0.78), but at the cost of proportionality. Conversely, lower $\beta$ values maintain significantly higher PPA levels than baselines. Furthermore, the framework demonstrates superior robustness; its PBM scores are markedly lower than those of RLHF, indicating increased resistance to strategic manipulation. On the Alpaca-GPT4 dataset, the proposed method successfully represented minority styles (PPA of 0.5012) that standard methods like DPO largely ignored (PPA of 0.3786).
Conclusion
This work provides a robust theoretical and empirical foundation for pluralistic AI alignment. By moving beyond simple majority aggregation, the proposed population-proportional framework offers a principled way to ensure AI systems respect the breadth of human values. Future work will address the challenges of measuring proportionality in settings where group identities are implicit and extending the axioms beyond top-ranked preferences.
Summary updated