Abstract
Prevailing methods for assessing and comparing generative AIs incentivize responses that serve a hypothetical representative individual. Evaluating models in these terms presumes homogeneous preferences across the population and engenders selection of agglomerative AIs, which fail to represent the diverse range of interests across individuals. We propose an alternative evaluation method that instead prioritizes inclusive AIs, which provably retain the requisite knowledge not only for subsequent response customization to particular segments of the population but also for utility-maximizing decisions.
Introduction
Current alignment techniques for generative AI, particularly Reinforcement Learning from Human Feedback (RLHF), often rely on head-to-head comparisons to determine model "helpfulness." This approach implicitly assumes a "representative individual" whose preferences the AI should mirror. However, human preferences are inherently heterogeneous. By optimizing for a single winning response, current methods incentivize "agglomerative" models that suffer from distribution collapse, losing the diverse perspectives captured during initial pretraining on the web.
The Problem of Agglomerative AI
Agglomerative models strive for a single optimal response to any prompt, essentially aiming to be a Dirac delta distribution. This "monomaniac" selection process renders models incapable of serving specific sub-populations or assisting users who require exploration of multiple valid options. For instance, in creative writing or medical diagnosis, a single "average" answer may be less useful than a distribution of plausible alternatives. Current fine-tuning practices actually corrupt the knowledge about human diversity that models accumulate during pretraining.
The Inclusive Objective
To address this, we propose an "inclusive" objective. Unlike standard scoring—which simply counts wins—the inclusive objective uses a log-probability-based score. It judges an AI based on the probability it assigns to whatever response a human rater prefers, even if that response was not the one the AI actually sampled in that specific trial. This rewards models for maintaining a distribution of responses that reflects the full spectrum of human interests.
Didactic Example
Consider a simplified scenario with two possible responses and three trials to illustrate the mathematical distinction between these approaches.
In this example, AI B consistently produces the most frequent "winning" response (the agglomerative approach), while AI A provides a variety of responses (the inclusive approach). Under standard benchmarks, AI B would be favored because it wins more often. However, AI A is more inclusive as it retains the knowledge necessary to satisfy different segments of the population through sampling.
Theoretical Foundations
We model human preferences using the Multinomial Logit Model, a classic tool from economics. Our theoretical analysis yields several key insights:
- The RLHF Paradox: Standard win-rate metrics favor deterministic models (Theorem 2), which leads to sub-optimal downstream decision-making. Deterministic is not optimal when preferences are diverse.
- Optimality of Inclusion: Under the multinomial logit model, the "maximally inclusive" AI is the unique optimizer of the proposed inclusive objective (Theorem 3).
- Utility Maximization: Inclusive AIs allow decision-makers to reach near-optimal results through repeated sampling. Conversely, agglomerative AIs can lead to permanent gaps in decision quality. In our didactic example, the agglomerative AI's refusal to show alternative options led to a permanent 33% gap in decision quality that no amount of extra sampling could fix (Theorems 4 & 5).
Conclusion
Inclusive AI offers a path toward models that better represent human diversity without sacrificing utility. While this work is primarily theoretical, it provides a robust mathematical framework for rectifying the weaknesses of current RLHF practices. Future research will focus on translating these principles into practical, large-scale training pipelines and addressing the challenge of accessing model log-probabilities during evaluation.
Summary updated