Standard machine learning evaluation often relies on a "winner-take-all" philosophy: a model is considered accurate if its prediction matches the majority label. However, human opinion is rarely a monolith. On subjective topics—ranging from the offensiveness of a comment to the "correct" response to a moral dilemma—humanity exists as a distribution of perspectives rather than a single consensus point.
Researchers are shifting toward Distributional Alignment, a paradigm where Large Language Models (LLMs) are trained and evaluated based on their ability to replicate the full statistical spread of human opinions. Instead of predicting the average score, these models aim to mirror the diversity, uncertainty, and disagreement inherent in human populations.
The Problem of Preference Collapse
The dominant method for training modern AI, Reinforcement Learning from Human Feedback (RLHF), suffers from a phenomenon known as preference collapse. As Jiancong Xiao and colleagues observe in Theoretical Tensions in RLHF, if 60% of humans prefer Answer A and 40% prefer Answer B, a perfectly optimized RLHF model will not provide a 60/40 split. Instead, it will converge on Answer A nearly 100% of the time.
This "winner-takes-all" behavior effectively silences minority perspectives. To counter this, Xiao and Ziniu Li propose Preference Matching (PM) RLHF in their 2024 work. The goal is to force the ratio of a model's output probabilities to align with the ratio of human preference scores, ensuring the model's stochastic nature reflects the underlying diversity of human feedback.
Measuring the Gap Between Model and Population
To determine if a model truly "understands" a population, researchers use divergence metrics to compare the model’s internal probability distributions against empirical human survey data.
- Kullback–Leibler (KL) Divergence: Used by Ricardo Dominguez-Olmedo and Moritz Hardt to measure the statistical distance between model responses and demographic data in their 2024 study. They found that many models, when corrected for arbitrary labeling biases, trend toward uniform randomness rather than human-like distributions.
- Jensen-Shannon (JS) Divergence: Preferred for its symmetry, this metric is used by Yu Liang and Christopher M. Homan to evaluate Population-Level Learning (PLL) in toxicity detection. By capturing the shape of annotator disagreement, models can provide a more nuanced reflection of what a community finds offensive.
- Wasserstein Distance: This metric is particularly useful for ordinal data (like Likert scales). Elle Michelle Yang uses it to audit Reward Models, revealing that the "judges" used to train LLMs often harbor significant sociodemographic biases.
The Knowledge-to-Simulation Gap
A recurring finding across this research is that LLMs often "know" what a distribution looks like without being able to "be" that distribution. Carlos Guestrin and Nicole Meister identify a knowledge-to-simulation gap: a model can accurately describe a group's opinion distribution in text (e.g., "40% of people agree with X"), but when asked to simulate individuals from that group through sampling, it fails to replicate that same statistical spread.
Similarly, Kathryn Davidson and Hayley Ross show that while LLMs can generalize linguistic rules, they struggle to mimic the full human distribution of adjective-noun inferences, often appearing more "certain" and biased toward positive membership than the average person.
Applications in Subjective Domains
The shift toward distributional alignment is proving essential in fields where "ground truth" is elusive:
- Common Sense: Michael Boratko and Andrew McCallum argue that common sense is a probability distribution of plausible outcomes. Their PROBEVAL metric measures how well a model's answer distribution matches human consensus.
- Moral Reasoning: Giuseppe Russo and Dirk Hovy address the "pluralistic moral gap" by using Dynamic Moral Profiling, which synchronizes a model's judgments with the statistical frequency of diverse human viewpoints.
- Cultural Values: Srishti Yadav and Daniel Hershcovich use JS Similarity to quantify cultural alignment in multimodal models, finding that alignment is often topic-dependent and does not automatically improve with model scale.
By treating human opinion as a distribution to be matched rather than a single label to be predicted, this research direction seeks to create AI that is not just "accurate" by majority standards, but representative of the complex, pluralistic societies it serves.