Replica Consistency: Testing Epistemic Uncertainty With an Equilibrium Identity

A model's uncertainty cannot be checked input by input: for any single test image we see only whether the prediction was right or wrong, never the true probability of error. Statistical physics faces the same problem with hidden states and has a partial answer. A system at equilibrium must satisfy exact identities that relate quantities we can measure to quantities we cannot. Well-specified Bayesian inference is formally such an equilibrium, and the Nishimori identity from spin-glass physics then says that the hidden truth is statistically interchangeable with a sample from the posterior. We propose to use this interchangeability, applied to pairs of model replicas (two ensemble members, two dropout masks, two training iterates), as a test of epistemic uncertainty. The test needs no uncertainty labels. It needs only ordinary labels, and only to estimate a single average.

Stated in two sentences: if a model's uncertainty is honest, two of its replicas should agree with each other exactly as often, on average, as one replica agrees with the true label. When replicas agree with each other more than with the truth, as happens when an ensemble is confidently wrong because the model class is misspecified, the size of that excess measures what ensemble variance misses. It also tells us how much to widen the epistemic spread.

Core Claim

Let $p_\theta(k \mid x)$ be the class probabilities of one replica with parameters $\theta$, and let $\theta, \theta'$ be two independent draws from the method's epistemic distribution. Assume a well-specified Bayesian posterior, meaning the true parameters $\theta^*$ are drawn from the prior and labels are generated from $p_{\theta^*}$. The Nishimori property then makes $\theta^*$ exchangeable with $\theta$, which implies

$$\mathbb{E}_{x,y}\,\mathbb{E}_{\theta}\big[p_\theta(y \mid x)\big] \;=\; \mathbb{E}_{x}\,\mathbb{E}_{\theta,\theta'}\Big[\textstyle\sum_k p_\theta(k \mid x)\,p_{\theta'}(k \mid x)\Big].$$

The left side, the replica–truth overlap, needs labels. The right side, the replica–replica overlap, needs none. We call their ratio on a subset $S$ of inputs the effective temperature $T_{\text{eff}}(S)$, defined as replica–replica overlap divided by replica–truth overlap. It equals 1 at "equilibrium."

We add a second identity that uses the averaged predictive $\bar p = \mathbb{E}_\theta p_\theta$ in place of two independent replicas. Holding both identities at once separates two failures. One is that each replica is individually miscalibrated, an aleatoric failure. The other is that replicas are too similar to one another relative to the truth, an epistemic or misspecification failure. The single-replica version is a known consequence of ordinary calibration. The cross-replica version is new as a diagnostic, and it has three useful properties. It targets exactly the failure mode that ensembles cannot see. It can be computed per region of input space. It is a scalar average, so it can be estimated to precision $\epsilon$ from $O(1/\epsilon^2)$ labels by Hoeffding's inequality for bounded means, with no binning.

The borrowed physics is the effective temperature. In active matter, when the fluctuation–dissipation theorem is violated, the violation is summarized by effective temperatures that can differ from one degree of freedom to another (Odd dynamics of passive objects in a chiral active bath; Hargus et al.; Mugnai et al.). Our prediction is the learning analogue: a misspecified model shows several effective temperatures across input regions, and the regions where $T_{\text{eff}} > 1$ are the regions where it is confidently wrong.

Position Relative to Prior Work

Several papers motivate the proposal by documenting the gap. DEUP: Direct Epistemic Uncertainty Prediction shows that ensemble and posterior variance collapse under misspecification, and fixes this by training a second network on held-out errors. Rethinking Aleatoric and Epistemic Uncertainty argues that the field conflates what a model believes with how it performs. Our identity measures exactly the gap between those two quantities, but it does so with a scalar per region rather than with a learned per-input regressor.

Expectation consistency for calibration of neural networks supplies the method. It uses the Nishimori identity to choose one post-hoc temperature so that average confidence matches accuracy. That is a single-replica, first-moment condition with one adjustable parameter, and it cannot touch the epistemic spread. We extend it to the two-replica overlap, to two temperatures, and to per-region diagnosis.

Legitimate ground-truth-free metrics for deep uncertainty classification scoring shows that ranking metrics (UQ-AUC, UQ-C-index) are tied to the latent misclassification probability. Those metrics evaluate ranking. Ours evaluates the size of the epistemic spread, so the two are complementary.

Two papers constrain the design. Measuring Uncertainty Calibration finds that certified calibration bounds near 0.02 need about $10^7$ samples. Verified Uncertainty Calibration shows that binned error estimates understate the error of continuous calibrators. Both push us toward scalar identities rather than per-bin estimates, and we use the second paper's debiased ECE estimator for evaluation.

Finally, we turn Fluctuation-dissipation relations for stochastic gradient descent and Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent into a testable prediction. At a finite learning rate $\lambda$, SGD's stationary covariance obeys $\Sigma K + K\Sigma - \lambda K\Sigma K = \lambda C$ rather than a Gibbs law. Iterates of SGD used as replicas should therefore violate the identity in a way that depends on the learning rate, even when the model is well specified.

One related observation should be named even though it is not in our file. Practitioners have reported empirically that disagreement between ensemble members tracks test error. Our identity states when that relationship should hold exactly, and it treats departures from it as the signal rather than as noise.

Research Program

The first stage is theory in a controlled setting. We use Gaussian-mixture classification, where the Bayes classifier is known, the setting in which Expectation consistency did its replica analysis. There we derive $T_{\text{eff}}$ in closed form for three cases: a well-specified linear model, a misspecified model (a linear fit to a curved boundary), and label noise. The goal is to prove that $T_{\text{eff}} > 1$ tracks the misspecification error and that the two identities separate noise from misspecification.

The second stage builds the tool. We will write estimators for both overlaps on each subset $S$, obtained by clustering penultimate-layer features, and attach Hoeffding confidence intervals to them. We will also build a two-temperature calibrator. It has one temperature on each replica's logits and one multiplier that widens or shrinks the spread between replicas, for example by interpolating each member's logits toward or away from the ensemble mean. The two parameters are solved so that both identities hold on a validation set. For local correction, the spread multiplier may vary by region.

The third stage is an empirical study. We apply the tool to four families of replicas: deep ensembles, MC dropout, SGD iterates collected at several learning rates, and variational inference. For SGD, we check Yaida's stationarity metric first, so that non-stationarity is not mistaken for misspecification.

Baselines

The naive reference point is the uncalibrated model, with its raw softmax confidences and ensemble variance. It fixes the floor for every metric.

The strongest calibration competitor is expectation consistency (Expectation consistency), alongside ordinary temperature scaling. On CIFAR-100 with 10% label corruption, the EC paper reports that EC "reduced the ECE to 5.9%, whereas TS only reached 9.6%." All methods receive the same trained ensembles and the same validation split with the same number of labels. Beating EC means lower debiased ECE and negative log-likelihood under corruption and shift, with the gain concentrated in regions our diagnostic flags.

The strongest competitor for epistemic uncertainty is DEUP (DEUP). It reports a Spearman correlation between uncertainty and error of 0.426, against 0.381 for deep ensembles on CIFAR-10. DEUP will receive exactly the same held-out labels that our method uses to estimate replica–truth overlaps, and its error predictor will get the same tuning budget. Beating it means higher Spearman correlation and higher UQ-AUC at equal label budgets, especially at small budgets of 500 to 2,000 labels. That is where a scalar-per-region estimate should win over a per-input regressor.

We will also run the stochastic variational inference model trained with the accuracy-versus-uncertainty loss (SVI-AvUC, Improving model calibration with accuracy versus uncertainty optimization) on CIFAR-10-C. That paper reports lower ECE than ensembles and TS but gives no numbers we can quote here, so we will rerun it.

Two ablations isolate the key ingredient. The first is a one-identity version that matches only the single-replica overlap, which is EC-like. The second is a global version that uses one spread multiplier for all regions. If neither ablation loses to the full method, the cross-replica, per-region identity is not doing the work.

Why not just…?

Why not just use deep ensembles with temperature scaling? Ensemble variance is precisely the quantity that vanishes when all members share the same wrong function. DEUP's Ackley and CIFAR results document this (DEUP), and in protein engineering ensembles were accurate but poorly calibrated (Benchmarking uncertainty quantification for protein engineering). Temperature scaling rescales each member but cannot widen the spread between members. Our misspecified-mixture experiment tests this directly, because the true misspecification error is known there.

Why not just fit two temperatures by minimizing validation log-likelihood? This is the real threat, because it uses the same two knobs without the physics. The evidence against it is indirect: likelihood-fitted temperature scaling degrades under label noise, where EC does not (9.6% versus 5.9% ECE on corrupted CIFAR-100). We will run the two-temperature likelihood fit head to head with our method. If it matches us on calibration, we will say plainly that the calibrator's contribution is small, and the claim will narrow to the diagnostic. The per-region $T_{\text{eff}}$ map, with its confidence intervals and its label-free half, is something likelihood fitting does not provide.

Why not just train DEUP's error predictor? DEUP learns a per-input function from held-out errors, so it needs enough labels to fit a network, and it can overfit when labels are scarce. Our estimate is a bounded mean per region with a guaranteed Hoeffding rate. The comparison of label budgets settles which is better in each regime. We expect DEUP to win when labels are abundant, and we will report it if that happens.

Why not just report ECE or UQ-AUC? ECE depends on the choice of bins and is sample-hungry (Measuring Uncertainty Calibration). UQ-AUC measures ranking, not the magnitude of the spread (Legitimate ground-truth-free metrics). An ensemble can rank perfectly and still be uniformly too narrow. We will construct such cases and show that $T_{\text{eff}}$ detects them while UQ-AUC does not.

Proposed Experiments

Synthetic mixtures. On 2D and high-dimensional Gaussian mixtures with a known Bayes classifier, we will vary misspecification (the degree of the model class) and label noise. The measure is the correlation between per-region $T_{\text{eff}}$ and the true excess error of each region. This is the decisive check. Legitimate ground-truth-free metrics reports 94% to 99% correlation with hidden ground truth for its ranking metrics on such data, which gives us a bar to match for a spread-magnitude measure.

Image classification. On CIFAR-10 and CIFAR-100 with 0%, 10%, and 30% label corruption, and on CIFAR-10-C and ImageNet-C across corruption severities, we will compare debiased ECE (uncertainty-calibration package), negative log-likelihood, Spearman correlation, and UQ-AUC against the baselines above. We will vary the label budget from 250 to the full validation split.

Separating the two kinds of uncertainty. On CIFAR-10-H, human soft labels give an aleatoric reference. The within-replica identity should track human disagreement, and the cross-replica excess should not. On ModelNet40-E, the physically annotated noise level σ(x) plays the same role for point clouds.

SGD replicas. We will predict and test that $T_{\text{eff}}$ for SGD iterates departs from 1 as the learning rate grows, in the direction implied by the $-\lambda K\Sigma K$ term, while ensembles of independently trained models at the same learning rate do not.

Falsification. On in-distribution CIFAR-10 with a well-specified large model, if $T_{\text{eff}}$ departs from 1 as much as it does on corrupted and shifted data, then the identity is not diagnosing misspecification and the central claim fails.