Which Grade Boundary Is Uncertain? Threshold-Level Disagreement Between Unlike Models

Many prediction targets are ordered grades: diabetic retinopathy stages 0 to 4, a person's age in years, a 5-point toxicity rating. A common way to model such grades is ordinal binary decomposition. Instead of predicting one grade, the model answers a chain of yes/no questions of the form "is the grade at least $k$?", one per boundary between neighbouring grades. This note proposes using that chain as a place to measure disagreement. We pair two models that see images differently, such as a large vision transformer and a small convolutional network. Each is trained to answer the boundary questions. We then measure, one boundary at a time, how much the two disagree beyond what each is individually unsure about. The result is an uncertainty signal that states which boundary is in doubt ("stage 1 vs. 2", not just "uncertain"). It grows with how far apart the models' answers are, so a grade-0 vs. grade-4 split counts as more dangerous than a grade-1 vs. grade-2 split. It also separates "the case is genuinely ambiguous" from "the models have not learned this kind of input."

This matters most where a wrong grade triggers the wrong clinical action and where the test data come from a different hospital or camera. Existing ordinal uncertainty measures are computed from a single model's output, which cannot tell ambiguity apart from ignorance. Existing ensemble disagreement measures treat classes as unordered. We want the missing combination, delivered at close to single-model cost.

Core Claim

Uncertainty quantification in ordinal classification: A comparison of measures shows how to "lift" binary uncertainty measures to ordinal targets:

  • decompose the predicted distribution into the cumulative yes/no splits;
  • score each split with a binary measure;
  • sum the scores.

It shows this beats entropy for rejecting predictions, because entropy ignores label order. That paper uses single models and explicitly leaves the split into epistemic uncertainty (not knowing) and aleatoric uncertainty (inherent ambiguity) to future work. Separately, Asymmetric Duos: Sidekicks Improve Uncertainty and Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification show that disagreement between architecturally different models is a cheap, strong epistemic signal. Both evaluate it only on unordered classes.

Our claim joins these results. Let member $m$ of an $M$-member ensemble output cumulative probabilities $p_{mk} = P_m(y > k \mid x)$ for boundaries $k = 1,\dots,K-1$. Let $\bar p_k$ be their average and $H$ the binary entropy. The per-boundary epistemic score is the binary mutual information (the entropy of the average minus the average of the entropies):

$$E_k(x) = H(\bar p_k) - \frac{1}{M}\sum_{m} H(p_{mk}), \qquad E(x) = \sum_{k=1}^{K-1} E_k(x).$$

The remaining term, $A(x)=\sum_k \frac{1}{M}\sum_m H(p_{mk})$, is the aleatoric part. Summing over boundaries gives the distance sensitivity Haas and Hüllermeier want. When two members' predicted grades lie far apart, they disagree on every boundary in between, so $E$ accumulates across many terms. A related identity sharpens the intuition: for two members, $\sum_k |p_{1k}-p_{2k}|$ is exactly the Wasserstein-1 (earth-mover) distance between their grade distributions.

We predict three things for this score computed on a heterogeneous duo:

  1. It ranks cases for abstention better, in ordinal error, than single-model lifted measures and than homogeneous ensembles of equal compute, especially under dataset shift.
  2. The profile $E_k$ localizes errors: the boundary with the largest $E_k$ is the one the wrong prediction crosses.
  3. $A$ tracks human annotator spread while $E$ does not.

We deliberately do not claim a new fusion rule. If each member uses a rank-consistent head of the kind introduced in Rank consistent ordinal regression for neural networks with application to age estimation (CORAL: shared weights, ordered biases), then plain averaging of cumulative probabilities is already monotone. The "rank inconsistency across heterogeneous members" gap is therefore solved by the simplest possible rule, and we use that rule. The contribution is the uncertainty signal and the evidence about when architectural difference is what makes it work.

Research Program

We build duos of a large base model and a sidekick from a different architecture family, costing roughly 10–20% of the base model's compute, following the recipe in Asymmetric Duos.

Every member gets a CORAL head. Where annotator distributions exist, members are trained with the soft cumulative loss of Reducing Supervision Uncertainty Induces Model Miscalibration. That paper found such training calibrated at temperature one, whereas hard labels needed temperatures above one.

Members are fused by averaging cumulative probabilities after fitting one temperature per member on validation data. This is the static learned weighting of Asymmetric Duos moved into cumulative space. We explicitly avoid stacking. Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap found that logistic-regression stacking had the worst log-loss and calibration among ensembles.

The analysis has three parts. First, we compare $E$, $A$, the per-boundary profile $E_k$, and Wasserstein disagreement against alternative scores. Second, we measure error correlation between members; the Q-statistic used in the tabular paper serves as a diagnostic. This tells us whether a given pairing actually differs or has hit the "diversity ceiling" that paper reports. Third, we test whether $E_k$ can drive boundary-specific referral: send a retinal image to a human only when the uncertain boundary is one that changes treatment.

Baselines

Naive reference points. The first is a single model with a plain softmax over grades, scored by entropy. The second is the same model scored by the lifted binary-decomposition measures of Haas and Hüllermeier. The second is the stronger of the two, since that paper reports it beats nominal measures in rejection quality. Beating it isolates the value of adding a second model at all.

Strongest competitor. The main rival is a homogeneous deep ensemble of the base architecture with different seeds, scored with the identical threshold-wise mutual information. This is the fair heterogeneity test, because only the source of diversity differs. We run it at two sizes:

  • Equal compute: two base-architecture members, so the homogeneous side gets more compute than the duo.
  • Five members: Asymmetric Duos reports that its duos "matched the performance of a 5-model Deep Ensemble while using 4 times less compute" on nominal image tasks. We test whether that holds for ordinal targets.

All ensembles use the same training data, augmentation, and hyperparameter search budget per member. Temperatures are fit on the same validation split. Beating this baseline means a lower area under the risk–coverage curve, with risk measured as mean absolute error, on shifted test sets, at no more compute than the two-member homogeneous ensemble.

Strongest single ordinal models. For accuracy, the reference points are CORAL, with MAE 2.64 ± 0.02 on MORPH-2 in its paper, and AOR-DR, with top-1 accuracy of 80.3% on APTOS 2019 and 75.9% on DeepDR with a RETFound backbone. The duo need not beat AOR-DR on accuracy. It must not lose more than about a point while adding the uncertainty signal. For selective-prediction metrics on these datasets, we found no reported result on Lacuna for prediction rejection ratio or risk–coverage area, so we will produce these numbers ourselves for all methods.

Ablation of the key ingredient. The ablation keeps the heterogeneous duo fixed and replaces the threshold-wise score with the variance of the members' expected grades. If the per-boundary decomposition adds nothing over that scalar, the "which boundary" part of the claim fails.

Why not just…?

Why not just use a single ordinal model's lifted uncertainty? Haas and Hüllermeier show it already beats entropy. However, a single model's output cannot distinguish an ambiguous image from an unfamiliar one, and under hospital shift the unfamiliar case is the dangerous one. The experiment trains on APTOS 2019 and tests on DeepDR and Messidor. There, the single-model measure should degrade relative to the duo's epistemic term. If it does not, the ensemble is unnecessary and we will report that.

Why not just use a homogeneous deep ensemble? This is the real threat. Seed diversity may suffice on in-distribution age data, and the tabular study shows nominally different models can be nearly redundant. The equal-compute and five-member comparisons above settle it. We expect heterogeneity to matter mainly under shift, where same-architecture members tend to fail together. We will state the result per dataset rather than average it away.

Why not just take the variance of the members' predicted grades? It is simpler and also distance-aware. It misses cases where members agree on the mean grade but disagree in shape, for example one member confident at grade 2 and the other split between grades 0 and 4. It also cannot say which boundary is in doubt. The ablation measures whether this matters in practice. If it gives the same rejection curves, we keep the per-boundary profile only for localization and drop the claim that it ranks cases better.

Why not regress the grade as a number and ensemble that? Evidence in the file consistently favours ordinal modelling. Examples are CORF's FGNET MAE of 2.68 against 4.10 for mean-variance regression in Deep Ordinal Regression Forests, and NORDIC beating support vector regression in Noncrossing Ordinal Classification. We include a regression duo only as a sanity check.

Proposed Experiments

Retinal grading. We train on APTOS 2019 and test in-distribution, then on DDR, DeepDR and Messidor as shifted domains. We report:

  • accuracy, MAE, and quadratic weighted kappa;
  • per-boundary calibration error;
  • risk–coverage area and prediction rejection ratio with MAE as risk;
  • the fraction of errors of two or more grades caught at 10% abstention;
  • detection of shifted-domain images from $E$ alone.

Decisive check 1 (localization). Among misgraded images, we test whether the boundary with maximal $E_k$ lies between the true and predicted grade more often than chance.

Face age. We repeat the core comparisons on MORPH-2, CACD and AFAD, training on one dataset and testing across the others. This checks that the effect is not specific to medical images, and against CORAL's reported MORPH-2 MAE.

Decisive check 2 (aleatoric vs. epistemic). We use the Toxicity and Aggression Likert datasets, which carry annotator distributions. We compute Spearman correlations of $A$ and $E$ with annotator spread, and with error on a held-out topical shift. The claim holds if $A$ correlates with annotator spread and $E$ does not, while $E$ correlates with shifted-domain error more than $A$ does.

Decisive check 3 (heterogeneity). We plot member error correlation against rejection gain across pairings: ViT+CNN, CNN+CNN, and ViT+ViT. Architectural difference should predict the gain, and seed difference should not.

If the duo fails to beat the equal-compute homogeneous ensemble on the shifted retinal and age tests, the heterogeneity part of the proposal is refuted. The threshold-wise epistemic decomposition would then remain a smaller, still useful contribution.