When several expert pathologists or ophthalmologists read the same image, they often disagree. That disagreement carries clinical information: a biopsy that six pathologists split between two grades should be handled differently from one they agree on. Models that predict a distribution over diagnoses, rather than one answer, are meant to reproduce this spread. Nobody has checked whether they still do so after deployment to a new hospital or scanner. This proposal treats a model's per-case predictive distribution as a generative model of the expert panel and scores it on two separate axes, as is done for image generators. Precision asks whether the model puts mass only on readings some expert would plausibly give. Recall asks whether it covers the minority readings experts actually give. We then measure how each axis degrades under controlled and natural domain shift, and for different pretrained backbones.
The central claim is concrete and testable. Under domain shift, multi-rater models lose recall of disagreement much faster than they lose precision: they collapse onto one reading and drop plausible alternatives. This would be the label-distribution version of the "confidently wrong" failure documented for referral systems. Measuring this honestly also requires an estimator we do not yet have. Panels contain only 6 to 20 raters per case, so the empirical panel distribution is badly undersampled. A reading that 10% of experts would give goes unseen in a 6-person panel with probability $(1-0.1)^6 \approx 0.53$, since each rater independently misses it with probability $0.9$. Without correction, a model that correctly covers such a reading is scored as hallucinating. The work therefore has two parts: a finite-panel estimator of disagreement precision and recall whose bias we validate against large panels, and the shift study that estimator makes possible.
What the literature already establishes
Three lines of prior work motivate the proposal without answering it.
The multi-rater modeling line builds models of the panel itself. Pionono gives each rater a Gaussian latent style. The Latent Doctor Model uses a conditional variational autoencoder over rater support sets. Virtual expert panels train one network per pathologist. MRNet and multi-rater prompting (PU-Net) condition a segmentation network on rater identity. All five are evaluated in-distribution, or in PU-Net's case across RIGA sub-cohorts only. All five are scored with Dice, Cohen's κ, KL divergence, entropy correlation or ECE. None of these scores separates the two ways a predicted distribution can be wrong.
The domain-shift line supplies the failure mode we expect to find. Avoiding Catastrophic Referral Failures showed that models trained on EyePACS and deployed on APTOS make low-entropy errors, which breaks referral. Ouyang et al. repaired out-of-domain segmentation calibration with spatial temperature maps. Both treat labels as single, objective truths. They cannot ask whether shift erases legitimate disagreement, as opposed to merely increasing error.
The transfer-learning line supplies the backbones whose robustness we compare. RETFound was pretrained on 1.6M retinal images and evaluated across countries, but only against consensus labels.
The method comes from generative-model evaluation. Djolonga et al. define precision-recall frontiers through Rényi divergences to an intermediate distribution. Liu et al. bound the sample complexity of these frontiers, show that missing mass dominates at small sample sizes, propose add-1/2 and Good-Turing smoothing, and summarize the frontier as a bounded frontier integral. That paper also constrains the design. Its guidance that bins should scale as $k \approx n^{1/3}$ says that for continuous outputs such as masks, per-case quantization with $n = 6$ raters is hopeless. We therefore reduce masks to a clinically meaningful scalar (vertical cup-to-disc ratio, binned) rather than clustering mask embeddings. This follows the diagnosis-first view that segmentation disagreement matters through its diagnostic consequence. Fréchet-style scores and metric-space magnitude are less suited here: they compare whole populations of samples and would mix across-case variation with within-case disagreement.
Research Program
The first component is the estimator. For case $i$, let $P_i$ be the panel's distribution over $K$ diagnostic grades, observed only through $m_i$ rater labels. Let $Q_i$ be the model's predictive distribution, read directly from softmax heads or estimated from many samples for latent-variable models. We compute the Djolonga frontier between the smoothed estimate $\hat P_i$ and $Q_i$, summarize it by Liu's frontier integral, and report its precision and recall endpoints separately, averaged over cases.
Three corrections are compared. The first is add-1/2 smoothing of $\hat P_i$. The second is Good-Turing missing-mass estimation pooled across cases of similar panel entropy. The third is a hierarchical Dirichlet prior over grades, fit on training panels and shared across cases. We also compute bootstrap intervals over raters, so that we can state when two models' differences are resolvable at a given panel size.
Validation uses rater subsampling. On ISUP prostate grading (20 pathologists) and DEPP (about 11 per case), the full panel serves as reference. We subsample 3, 6 and 9 raters, and ask which estimator recovers the full-panel precision and recall, and the ranking of models, with the least bias.
The second component is the shift study. We train a grid of models: consensus softmax, Pionono, conditional LDM, PU-Net prompts, and on DEPP a virtual panel. These are crossed with backbones: from scratch, ImageNet-pretrained ResNet, and RETFound for fundus. Each model is then evaluated under two kinds of shift.
Natural shift comes from RIGA: train on MESSIDOR and BinRushed, test on Magrabia, with 6 raters throughout, and the multi-center portion of QUBIQ. Controlled shift applies graded stain perturbation, blur and compression at five severities to Gleason2019, ISUP and DEPP test sets. This yields curves of disagreement precision and recall against severity. Cases are stratified by full-panel entropy, because the hypothesis concerns the contested cases. We also include DEPP's "indefinite for dysplasia" cases, which the virtual-panel paper excluded, since they are where coverage matters most.
The third component tests whether existing repairs restore recall. We apply temperature scaling, Split Platt Scaling from the referral paper, and test-time-augmentation ensembling to the shifted models. If they restore average calibration but not recall on high-disagreement cases, that is the finding a clinician-facing system most needs.
Baselines
The naive reference is a consensus-trained classifier whose softmax is read as a distribution over panel readings, with and without temperature scaling on in-distribution validation data. This is what most deployed systems, including RETFound's heads, effectively do. Any multi-rater method that fails to beat it on recall under shift tells us something important. For context, RETFound reports diabetic retinopathy AUROC of 0.943, 0.822 and 0.884 across three validation sets (RETFound), but no rater-distribution result is reported.
The strongest existing distribution-modeling methods are Pionono and the conditional LDM. On Gleason2019, Pionono reports unweighted κ = 0.758 (Pionono) and the conditional LDM reports KL = 0.461 (Latent Doctor Model). On ISUP, the LDM's prediction entropy reaches Spearman 0.728 with panel entropy. On DEPP, the full virtual panel reports F1 78.2% versus 72.9% for consensus, with ECE 0.050 versus 0.183 (virtual expert panel). For PU-Net on RIGA, no reported Dice result was found on Lacuna. MRNet reports only a 1.2% soft-Dice gain for the optic cup.
Fairness requires several controls. All methods share the same backbone initialization within each backbone condition and the same data splits. Each gets an identical hyperparameter search budget, tuned on in-distribution validation only, so that no method sees shifted data. We first reproduce the published in-distribution numbers above to within noise before trusting the shift results.
Since this is mainly an evaluation proposal, "beating" a baseline happens at two levels. For the estimator, success means our corrected precision and recall from 6 raters track the 20-rater reference with lower bias and higher rank agreement than the uncorrected estimate and than per-case KL or entropy correlation. For the science, the central claim is supported if recall drops significantly more than precision as shift severity grows, concentrated in high-disagreement cases, with non-overlapping rater-bootstrap intervals.
The key ablation removes small-panel correction. If uncorrected and corrected estimates rank models identically at 6 raters, the estimator component is unnecessary and the work reduces to the shift study.
Why not just…?
Why not just report per-case KL divergence to the panel distribution, as the Latent Doctor Model does? KL is one number, and it becomes infinite or unstable when the empirical panel has zero mass on a grade the model predicts, which is exactly the small-panel regime. More seriously, forward and reverse KL are themselves a crude precision/recall pair, since they sit at the frontier's endpoints. This is a real threat. We therefore include smoothed forward and reverse KL as a baseline evaluator. If they separate collapse from hallucination as well as the full frontier in the subsampling validation, we will say so and recommend them. In that case the contribution is the finite-panel correction and the shift finding, not the particular divergence.
Why not just use calibration error against soft labels? ECE averages confidence against accuracy across cases. A model can be well calibrated on average while placing disagreement on the wrong cases or the wrong grades. The virtual-panel paper's ECE gain says nothing about whether minority readings on contested biopsies survive. Our entropy-stratified analysis directly tests whether ECE and recall diverge under shift.
Why not just correlate model entropy with panel entropy? Entropy ignores which grades carry the mass. A model split between grades 3 and 5 matches the entropy of a panel split between 3 and 4. We will construct synthetic predictors with matched entropy but swapped grades and show that entropy correlation cannot distinguish them while recall can.
Why not just recalibrate with post-hoc temperature or Split Platt Scaling under shift? Those methods are repairs, not measurements, and they rescale confidence along one axis. A single temperature cannot reintroduce a dropped grade without also flattening agreed-upon cases. We run them as treatments in the third component. If they do restore recall on contested cases, that is a positive and useful result.
Proposed Experiments and Decisive Checks
The first decisive check is estimator validity. On ISUP and DEPP, with 6-rater subsamples drawn 200 times per case, we measure the bias and rank correlation of each estimator against full-panel precision and recall. Synthetic predictors with injected collapse (mass moved to the modal grade) and injected hallucination (mass moved to grades no rater gave) test whether the estimator assigns each failure to the correct axis.
The second check is the shift curve. We plot recall and precision against perturbation severity on Gleason2019, ISUP and DEPP, and report the natural-shift gap from MESSIDOR and BinRushed to Magrabia on RIGA binned cup-to-disc ratio. The hypothesis fails if precision and recall degrade at similar rates, or if degradation is not concentrated in high-entropy cases.
The third check is the backbone comparison on RIGA. Does RETFound initialization reduce the Magrabia recall drop relative to ImageNet and scratch, holding the multi-rater head fixed? A null result here would be informative given RETFound's cross-country consensus robustness.
The final check asks whether existing recalibration restores recall on contested cases or only average ECE. We report it alongside the clinical consequence: how many shifted contested cases would be referred under an entropy-threshold policy before and after shift.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling
- Multi-rater Prompting for Ambiguous Medical Image Segmentation
- Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- A foundation model for generalizable disease detection from retinal images
- Precision-Recall Curves Using Information Divergence Frontiers
- Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Metric Space Magnitude for Evaluating the Diversity of Latent Representations