Aligning Model Uncertainty with Human Disagreement Distributions

60 concepts44 papers1 proposal

In traditional machine learning, evaluation is straightforward: a model makes a prediction, and we compare it to a single "gold standard" ground truth label. If the model matches the label, it succeeds; if not, it fails.

However, this "winner-take-all" paradigm breaks down when applied to real-world, subjective, or ambiguous tasks. Whether determining if a comment is offensive, identifying an emotion on a face, or predicting the next word in an open-ended sentence, humans themselves rarely achieve perfect consensus.

When we collapse a diverse set of human annotations into a single majority-vote label, we discard a massive amount of signal. A growing body of research argues for a perspectivist paradigm shift: instead of forcing models to mimic an artificial consensus, we should train and evaluate them to match the full, empirical distribution of human opinions.

The Fallacy of the Single Gold Label

To understand why traditional evaluation fails, consider a text snippet that 60% of annotators rate as "neutral" and 40% rate as "offensive."

Under a standard majority-vote framework, the ground truth becomes "neutral." A model that outputs a 100% confident prediction for "neutral" receives a perfect accuracy score. Conversely, a model that accurately captures the community's division by predicting a 60/40 split is mathematically penalized for being "unconfident."

As argued by Fleisig et al., collapsing these labels silences minority perspectives and misrepresents the inherent ambiguity of the data. Uma et al. demonstrate that training models on the full distribution of human annotations—rather than a single hard label—allows systems to better navigate real-world nuance. When a task has low consensus, automated systems that try to force a single judgment often produce overconfident, low-entropy outputs that fail to reflect human reality (Revel et al.).

To fix this, researchers treat human annotations as a probability vector (or "soft label") where each element represents the proportion of human raters who chose that option. The goal then shifts from maximizing top-1 accuracy to minimizing the distance between the model's output probability distribution and the empirical human distribution.

Traditional Target (Hard Label):  [1.0,  0.0,  0.0]  <-- Information lost
Distributional Target (Soft Label): [0.6,  0.3,  0.1]  <-- Captures human disagreement

Quantifying Alignment via Distributional Distance

How do we mathematically measure whether a model's internal uncertainty mirrors human disagreement? Rather than using categorical metrics like accuracy or F1-score, this paradigm leverages statistical divergence and distance measures.

Jensen-Shannon Divergence (JSD)

Jensen-Shannon Divergence is a symmetric, bounded metric (ranging from 0 to 1 when using base 2) that measures the similarity between two probability distributions. Because it does not blow up to infinity when a model assigns a zero probability to a rare human label, it is highly favored for evaluating linguistic and semantic ambiguity.

For instance, Nie et al. introduced the ChaosNLI dataset—collecting 100 human annotations per example—and used JSD to reveal that state-of-the-art models fail to capture human-like uncertainty on ambiguous inference tasks. Similarly, Liang et al. utilized JSD to evaluate population-level learning in toxicity detection, ensuring that the shape and overlap of the entire opinion distribution are accounted for.

Kullback-Leibler (KL) Divergence

KL Divergence measures the information lost when using a model's predicted distribution to approximate the true human distribution. Boratko et al. applied KL divergence to evaluate open-ended commonsense reasoning, demonstrating that common sense is better represented as a spectrum of plausible outcomes rather than a single correct answer. By optimizing for lower KL divergence, models are rewarded for assigning appropriate probability mass to the "long tail" of valid human interpretations (Li).

Total Variation Distance (TVD) and L1 Distance

Total Variation Distance represents the maximum possible difference between the probabilities that two distributions assign to the same event. It provides an intuitive, absolute measure of the gap between model confidence and human rater frequency. Lan et al. and Baan et al. use TVD to expose instances where a model exhibits extreme confidence in a single prediction while human annotators remain deeply divided.

The Calibration Paradox

A common counterargument is that we can fix overconfident models using standard calibration techniques like Expected Calibration Error (ECE) or temperature scaling. However, recent work has exposed a fundamental mathematical conflict between standard calibration and human distributional alignment.

Traditional calibration metrics assume there is one true label. ECE bins predictions based on the model's confidence and checks if that confidence matches the accuracy relative to that single label.

Baan et al. demonstrated that this creates a paradox: a model that perfectly predicts the true distribution of human uncertainty can be mathematically classified as miscalibrated by standard ECE. If a model correctly predicts a 50/50 human split on an item, its confidence is 50%. But if the evaluation framework forces a single majority-vote ground truth, the model is penalized for not being 100% confident in the majority choice.

Data Point: Human raters are split 50/50 between Class A and Class B.
Majority Vote Ground Truth: Class A

Model 1 (Overconfident): Predicts 100% Class A. 
-> Standard Accuracy: 100%
-> Standard Calibration (ECE): Perfect (Confidence 100% matches Accuracy 100%)
-> Distributional Alignment: Poor (Fails to capture the 50% human dissent)

Model 2 (Human-Aligned): Predicts 50% Class A, 50% Class B.
-> Standard Accuracy: 50% (on aggregate)
-> Standard Calibration (ECE): Penalized (Confidence doesn't match a 100% certain target)
-> Distributional Alignment: Perfect (Mirrors the true human population split)

As Xu et al. show, post-processing calibration techniques can make a model honest about its overall error rate, but they completely fail to capture the specific probability distributions of expert disagreement. Making a model statistically "calibrated" is not the same as making it reflect human value pluralism.

To bridge this gap, researchers are shifting toward instance-level, human-aligned calibration. Instead of adjusting probabilities to match aggregate accuracy, techniques are developed to explicitly align model entropy with empirical human rater variance (Falk et al.).

Why Distributional Alignment Works

Transitioning from point-estimates to distributional targets fundamentally changes what a model learns during training and how it is judged during evaluation:

  1. Robustness to Noise: Petkov demonstrated that modeling raw categorical ratings as a distribution using Discrete Choice Models outperforms standard regression. Teaching a model how and why humans disagree provides a smoother optimization landscape that prevents the model from overfitting to arbitrary majority-vote boundaries (Wu et al.).
  2. Perceptual Fidelity: In visual and auditory tasks, human perception is continuous. By training models on soft targets derived from score distributions rather than one-hot vectors, models learn the underlying statistical spread of human perception, leading to significantly higher fidelity in subjective domains like image quality assessment (You et al.) and facial expression recognition (Bryant et al.).
  3. Measuring Perceptual Caution: Distributional metrics like the Hellinger Distance allow us to evaluate an AI's decision to "abstain" or show caution. Lee et al. show that evaluating the full profile of uncertainty across multiple classes reveals whether a model's hesitation matches human-like caution in out-of-distribution scenarios.

By moving away from the fiction of a single consensus, distributional alignment metrics provide a mathematically rigorous framework for building AI that doesn't just predict an answer, but understands the collective weight of human opinion.

Key Foundational Work

Simulating the Behaviour of Older versus Younger Users when Interacting with Spoken Dialogue Systems (2008) — contributed 1 concept elements to this direction.
User simulation for spoken dialogue systems: learning and evaluation (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_distributional_alignment_metrics_67369816,
  title = {distributional alignment metrics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distributional-alignment-metrics/rdr_67369816d667462bbc0aa7b7f0b5ee4f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_distributional_alignment_metrics_67369816_v1,
  title = {distributional alignment metrics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distributional-alignment-metrics/rdr_67369816d667462bbc0aa7b7f0b5ee4f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake