Measuring Human–Model Disagreement as a Tension Between Two Posteriors

Many language tasks have no single right answer. When 100 people label whether one sentence implies another, they often split. The usual way to score a model on such tasks is to compare its predicted label probabilities with the human vote shares using a divergence such as Jensen–Shannon. That single number mixes three different things. The first is noise in the human target itself, because a handful of annotators is a poor estimate of the crowd. The second is the model's own ignorance. The third, the one we actually want, is whether the model's sense of how ambiguous the item is matches the humans'. The mixing has visible consequences. A model that simply does not know can look "well aligned" on ambiguous items by spreading its probability evenly. On ChaosNLI, uniform chance baselines sometimes beat RoBERTa-large under KL divergence (Nie et al.).

Cosmologists face the same structural problem when two experiments, such as the Planck cosmic microwave background map and the Dark Energy Survey, disagree about a parameter. They have built statistics for asking whether two posteriors are in genuine conflict, and those statistics are designed to be insensitive to the arbitrary prior. We propose to import that machinery. For each item, the model contributes a distribution over possible label distributions, estimated from an ensemble. The annotators contribute a second such distribution from their vote counts. We then ask, in units of sigma, whether these two "datasets" are in tension. That conversion to sigma is itself a known weakness in cosmology, because it assumes Gaussian posteriors. Our second contribution is to replace that assumption with an exact simulated null, which is cheap on the low-dimensional space of label distributions and can then be tested back on cosmological chains.

Core Claim

An alignment score should reward a model whose aleatoric estimate matches the human spread. Aleatoric uncertainty is the part of its uncertainty it attributes to the item being ambiguous. The score should neither punish the model for honestly reporting epistemic uncertainty (its own ignorance) nor reward it for that ignorance. It should also not penalize the model for disagreeing with a noisy five-annotator estimate. Treating the model and the annotators as two posteriors over the same item-level label distribution, and scoring their agreement with a prior-insensitive tension statistic calibrated by simulation, does all three. A Jensen–Shannon score, even one corrected for annotator noise, cannot. The concrete deliverable is a per-item tension score in sigma that has three properties. It is uniformly distributed for an oracle model whose belief generated the human votes. It is stable across choices of the smoothing prior. When computed from five annotators, it predicts the misalignment measured with all 100.

The Construction

Let $p$ be a point on the probability simplex over the $K$ labels, here $K=3$ for natural language inference. Let $\pi(p)$ be a shared prior, for example a symmetric Dirichlet with concentration $\alpha$.

The annotators supply the likelihood $L_h(p)$, the multinomial probability of their observed counts. The model supplies a second-order density $q(p)$, obtained by fitting a Dirichlet to the predictions of independently trained ensemble members. The spread of $q$ is the model's epistemic uncertainty. Its location, whether near a vertex or near the centre, is the model's aleatoric judgment of how ambiguous the item is. We turn $q$ into an implied likelihood $L_m(p) = q(p)/\pi_0(p)$ relative to a reference prior $\pi_0$.

Following Lemos et al. 2020, the Bayes ratio is

$$R = \frac{\int L_m L_h \pi \, dp}{\int L_m \pi \, dp \int L_h \pi \, dp}, \qquad \log S = \log R - \log I,$$

where $I$ is the information ratio built from the KL divergences between each posterior and the prior, and $S$ is Suspiciousness. Removing $\log I$ cancels most of the dependence on how wide $\pi$ is. That dependence is the "Achilles' heel" of $R$ documented in Lemos et al. 2021. On a two-dimensional simplex, every integral is exact by quadrature.

Cosmology converts $S$ to sigma by assuming $d - 2\log S \sim \chi^2(d)$, with $d$ the effective dimensionality. On the simplex, posteriors pile up against the boundaries whenever humans are near-unanimous, so the Gaussian assumption is plainly wrong there. We instead compute the null directly. We draw $p \sim q$, draw $n$ simulated annotator votes from $p$, recompute $\log S$, and read off the observed value's quantile. This null asks the question we care about: could these votes plausibly have come from a world the model believes in?

The cosmology papers supply the statistic and motivate the null correction: Lemos et al. 2020 and Lemos et al. 2021 both flag the Gaussian conversion as a limitation. Baan et al. motivates the whole effort. It shows that expected calibration error scores a perfect oracle at 0.25 against 0.14 for an ordinary model, and it proposes instance-level metrics (DistCE, EntCE) that still compare a single predicted vector to a single empirical vector. Our proposal differs from Baan et al. in one specific way: we treat both sides as uncertain. The separation of epistemic from aleatoric uncertainty follows the logic of CLEAR, which shows that conflating the two causes coverage failures. CLEAR works on scalar regression intervals and never touches distributional targets or priors. We reuse its central lesson, but not its method.

Research Program

The program has four parts.

First, we build the scorer: the Dirichlet-fitted ensemble density, the exact quadrature for $R$, $I$ and $S$, and the simulated null.

Second, we study prior sensitivity directly. For a grid of Dirichlet concentrations $\alpha$, we measure how much the item rankings change under $R$, under $S$ with a $\chi^2$ conversion, and under $S$ with the simulated null. The hypothesis is that $R$ swings strongly with $\alpha$ and $S$ does not, reproducing the cosmological finding in a setting where ground truth is available.

Third, we test whether the scores survive sparse annotation. That matters because dense 100-vote annotation is the cost barrier noted by Nie et al..

Fourth, we run the scorer back on cosmological chains to check whether the simulated-null idea changes reported tensions.

Baselines

The naive reference is a single fine-tuned RoBERTa scored by Jensen–Shannon divergence and DistCE against the empirical vote vector, following Baan et al.. The same model is also scored after temperature scaling, a post-hoc rescaling of the model's output logits by a single learned constant. The file reports that temperature scaling cuts ECE from 14.7 to 3.3 on ChaosNLI but improves DistCE by only about 15%, with RoBERTa at 74% accuracy (Baan et al.). No reported Jensen–Shannon value for RoBERTa on ChaosNLI was found on Lacuna, so we recompute every baseline ourselves on the same splits.

The strongest competitor is a noise-corrected divergence. It computes Jensen–Shannon divergence from the ensemble's mean prediction, then subtracts the expected divergence between an $n$-annotator sample and the 100-annotator distribution. That correction is a split-half noise ceiling, and it already handles target noise. It is fair to give this baseline the same ensemble and the same annotator subsamples as our method. Our method beats it if two things hold. Its scores computed at $n=5$ must rank items and models more faithfully against the $n=100$ reference. It must also correctly separate models whose misalignment is purely epistemic from those whose misalignment is aleatoric.

There are two ablations. The first, "decomposition without tension", scores only the ensemble's aleatoric component against the votes with Jensen–Shannon divergence. The second keeps $S$ but uses the $\chi^2$ conversion instead of the simulated null. A third ablation substitutes $R$ for $S$. All variants share one ensemble of five seeds, so the compute is identical.

Why not just…?

Why not just correct Jensen–Shannon divergence for annotator noise? This is the real threat. A noise ceiling fixes the target-noise problem, and it may match us on rank fidelity at small $n$. It cannot tell a confused model from an ambiguity-aware one. Both produce a flat mean prediction, and that flatness is exactly why uniform baselines beat RoBERTa under KL divergence in Nie et al.. The synthetic-injection experiment below decides this. If the noise-corrected score separates the injected cases as well as ours does, the tension machinery is unnecessary, and we would report that.

Why not just use the ensemble's expected-entropy (aleatoric) component and skip the tension statistic? That ablation separates ignorance from ambiguity. It still treats five votes as truth, and it has no notion of how surprising a given divergence is. The sparse-annotation experiment measures what the tension calibration adds beyond the decomposition alone.

Why not just use a Bayes factor with a Dirichlet prior? It is principled, but it is prior-volume dependent. Lemos et al. show that $R$ reads $\log R \approx 0$ on a mock KiDS shift where Suspiciousness finds 3.09σ (Lemos et al. 2020). The $\alpha$-sweep tests whether the same failure appears on the simplex.

Why not just temperature-scale? Temperature scaling applies one global stretch and cannot express item-specific ambiguity. Baan et al. show it barely moves DistCE and harms items that were already well aligned (Baan et al.).

Proposed Experiments

All experiments use ChaosNLI: 4,645 items with 100 votes each, drawn from SNLI, MNLI and αNLI (Nie et al.). The models are five-seed RoBERTa-large ensembles fine-tuned on SNLI and MNLI.

The first decisive check is the oracle null. We define the oracle's density $q$ as a tight Dirichlet around each item's 100-vote distribution, then score subsamples of $n \in \{3, 5, 10, 20\}$ held-out votes. A correct scorer gives roughly uniform p-values. We expect the $\chi^2$ conversion to deviate most on near-unanimous items. This directly tests the oracle paradox that ECE fails.

The second check is synthetic injection. We create controlled variants of the ensemble. One adds epistemic spread without moving the mean, by perturbing individual members. Another sharpens every member toward the majority label. A third mixes every member toward uniform. The scorer must flag sharpening and uniform-mixing as aleatoric misalignment, and must not flag added epistemic spread. We measure this with the area under the ROC curve for each scorer.

The third check is sparse-to-dense fidelity. We compute each score with $n=5$ and correlate it, using Spearman rank correlation, with the Jensen–Shannon divergence computed on the full 100 votes, at the item level and across training-set variants. Prior sensitivity is reported as the variance of item ranks across $\alpha \in [0.1, 10]$.

The final, exploratory check returns to cosmology. Using the public unimpeded chains, we approximate each dataset's likelihood in the shared parameters by a density fitted to its samples. We then build a simulated null by resampling, and compare the result with the reported χ²-converted tensions of 3.57σ for DES Y1 versus Planck ($d_G \approx 6.6$) and 3.27σ for SH0ES versus Planck ($d_G \approx 1.4$). Our prediction is that the low-dimensional $H_0$ tension barely moves and the curved, high-dimensional $S_8$ tension moves more. A null result here still leaves the language-side contribution intact.