Separating "Look Closer" from "Genuinely Ambiguous" in Pathology Disagreement

When pathologists disagree about a region of tissue, a model trained on their multiple labels learns to report uncertainty there. That uncertainty has two very different causes that current systems do not separate. Some of it exists because the model is looking at a downsampled image: fine cellular detail has been blurred away, and a sharper view would settle the question. The rest of it exists because the tissue is truly borderline, and even six experts at full magnification split their votes. A system that zooms in wherever it is uncertain wastes its high-resolution budget on the second kind. A system that reports all uncertainty as "ambiguous" hides the first kind, which is only an artifact of resolution.

We propose to measure these two parts separately and to learn to predict them. We would fit the same soft-label classifier on frozen image features at several magnifications, with added summaries of each patch's neighborhood. The gap between the low- and full-resolution predictions is the part a closer look can resolve. The uncertainty that remains at full resolution is the intrinsic part, and it should track real expert disagreement. A small regressor that sees only the low-resolution view would then learn to predict the first part. That regressor tells a multi-resolution system where zooming will actually change the answer and where it will not. This needs no annotator identities and no human re-reading, and it can be checked directly against the six-pathologist labels in Gleason2019.

Core Claim

Our claim is that uncertainty which can be resolved by resolution is predictable from the low-resolution view alone. We claim further that zooming ranked by this prediction beats zooming ranked by total uncertainty at a fixed budget. The target is a better match to the full multi-rater label distribution, not to the majority vote.

Let $p_s(i)$ be the predicted distribution over Gleason classes at location $i$ when the image is downsampled by factor $s$, with $s=1$ meaning native resolution. Define the resolvable part as

$$D_s(i) = \mathrm{KL}\big(p_1(i)\,\|\,p_s(i)\big),$$

which is how much the answer would change if we looked at full resolution. Define the intrinsic part as $H(p_1(i))$, the entropy of the prediction at full resolution. The zoom-value regressor $g_s$ sees only low-resolution features at $i$ and is trained to predict $D_s(i)$.

We make two falsifiable predictions. First, $H(p_1)$ should correlate with the entropy of the six pathologists' votes, while $D_s$ should add little to that correlation. Second, upgrading the top-ranked locations by $g_s$ to full resolution should reduce KL to the rater distribution faster than upgrading by $H(p_s)$.

Research Program

Features. We extract patch embeddings from a frozen backbone at downsampling factors 1, 2, 4, 8 and 16 on Gleason2019 tissue microarray images. The main backbone is ConvNeXt-Tiny, fine-tuned once on native-resolution training cores and then frozen, following the recipe of Reliable Dental Radiograph Diagnosis via Calibrated Hybrid Representation Learning. That paper froze a fine-tuned ConvNeXt and found that classical heads on its embeddings were well calibrated. It used hard labels on isolated crops, and we reuse only its decoupling recipe. We also test a ResNet50 and a Swin backbone, because Deep Spatial Context found that the amount of spatial structure captured varies across backbones.

Neighborhood features. Each patch embedding is augmented with the mean of its 8 nearest neighbors' embeddings, in the style of a spatial-lag-of-X term. We also test a distance-weighted variant modeled on the adjacent-region indices of Improvement of Moderate Resolution Land Use and Land Cover Classification by Introducing Adjacent Region Features, which raised SVM accuracy on Landsat 8 from 88.87% to 96.97%. Two papers set the design here. Deep Spatial Context supplies the design choice: it found strong spatial autocorrelation in 99.4% of slides and found that immediate neighbors ($k=8$) carry most of the context. It also constrains the design, because its spatial measures dropped on held-out data in over 86% of configurations. For that reason every spatial-feature result is reported on held-out cores, and it counts only if it survives there. The landsat paper also warns that large windows smear across class boundaries, so we keep $k$ small.

Head. At each scale we fit a multinomial logistic regression whose targets are the per-location empirical distribution of the six pathologists' labels. We fit it with soft-target cross-entropy and then apply temperature scaling. We chose it because it is convex and cheap enough to refit at five scales and several backbones. It also keeps the comparison across scales about the features rather than about optimization luck. The head is a tool, not the claim, and the end-to-end alternatives below are run against it.

Zoom-value regressor. $g_s$ is a gradient-boosted regressor on low-resolution features plus neighborhood features that predicts $D_s$. It is trained on training cores, where $p_1$ is available.

Optional reader study. Two pathologists grade a sample of regions at factors 1 and 8. This tests whether low-resolution disagreement in humans concentrates where $g_s$ is high. None of our computational claims depend on it.

Baselines

The naive reference is a majority-vote model: the same frozen features and logistic head trained on the consensus label. Its KL to the rater distribution and its entropy correlation show what distributional training buys at all.

The strongest existing distribution-matching methods are the Conditional Latent Doctor Model and Pionono. The Conditional Latent Doctor Model is a conditional variational autoencoder that learns annotator styles from support sets. The Latent Doctor Model for Modeling Inter-Observer Variability reports a KL divergence of 0.461 against the Gleason2019 panel for it, and a Spearman correlation of up to 0.728 between its predictive entropy and pathologist entropy on ISUP grading. Pionono models each rater as a Gaussian in a latent space. Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation reports an unweighted Cohen's kappa of 0.758 for it on Gleason2019.

Both operate at a single scale and use annotator identities. We run their public formulations on our exact data split and at our same downsampling factors, with the same hyperparameter-search budget in GPU-hours. We report KL both at matched resolution and at matched compute. Our distribution-matching goal is modest: to stay within noise of these baselines on KL and kappa at native resolution. We do not claim to beat them there. We would claim a win only on the multi-resolution budget curves.

The strongest end-to-end soft-label baseline follows Pathologist-like explainable AI for interpretable Gleason grading in prostate cancer. That work trained a U-Net on soft labels from 54 pathologists and reached Dice 0.713 versus 0.691 for hard labels. Its data differ from Gleason2019, so that number is context rather than a target. We train this model per scale with the same augmentations and budget.

For zooming, there are two baselines. The first is entropy-ranked zooming using $H(p_s)$, plus a deep-ensemble variant. The second is attention-ranked top-$k$ zooming from Diagnose Like a Pathologist (HAG-MIL; code at BearCleverProud/HAG-MIL). HAG-MIL zooms on suspicious regions and scores 0.946 AUC on Camelyon16, but against hard labels. For zoom policies under multi-rater evaluation, no reported result was found on Lacuna. We therefore run every policy at identical budgets of 5%, 10%, 25% and 50% of locations upgraded to native resolution.

The Foveation module of Foveation for Segmentation of Ultra-High Resolution Images learns a field-of-view versus resolution choice end-to-end. It beat top challenge entries by 13.1% on Grade 3 against a single target. We retrain it on soft targets using its public code as a learned-policy competitor.

Two ablations isolate the key ingredients. The first removes the neighborhood features. The second replaces $g_s$ with a regressor trained to predict total entropy.

Why not just…?

Why not just zoom wherever the low-resolution model is most uncertain? This is the main threat. If intrinsic ambiguity is rare, or is itself concentrated where low-resolution blur is worst, then total entropy ranks locations almost as well as $g_s$. The budget-curve comparison settles it directly. If entropy ranking matches $g_s$ within confidence intervals across budgets, the decomposition has no operational value, and we would report that. The scientific finding would still stand if $H(p_1)$ tracks human entropy while $D_s$ does not.

Why not just train an end-to-end soft-label network at each resolution? This could match or beat the linear head on KL, as GleasonXAI suggests soft-label deep models work well. That would not undercut the claim, because the decomposition is defined for any head. We run the full analysis with both heads. If the end-to-end head wins, we adopt it and report the cost of refitting at five scales.

Why not just use rater-modeling methods like the Latent Doctor Model or Pionono? These methods model who disagrees, not why. They need annotator IDs at training and a support set at inference, and they do not vary resolution. We compare against them at native resolution to make sure distributional quality is not sacrificed. They offer no zoom signal to compare against.

Why not just enlarge the patch instead of adding neighborhood features? A larger receptive field is the simpler way to add context, as in patch-based CNNs. We include a larger-crop variant at matched feature dimension. Given the held-out degradation reported by Deep Spatial Context, we expect both forms of context to shrink out of distribution. We report whichever generalizes better.

Decisive Checks

We use the Gleason2019 split of 188 training, 33 validation and 23 test images from Gleason Score Prediction using Deep Learning in Tissue Microarray Image, with per-location distributions over the six pathologists' labels.

The first check tests the decomposition. On test cores, we compute the Spearman correlation between human vote entropy and $H(p_1)$, and the partial correlation of $D_8$ given $H(p_1)$. The hypothesis holds if the first is substantial and the second is near zero.

The second check tests predictability. We ask whether $g_s$ predicts $D_s$ on held-out cores, measured by Spearman correlation and top-decile precision, with and without neighborhood features.

The third check is the operational one. We plot KL to the rater distribution against zoom budget for $g_s$, entropy, ensemble entropy, HAG-MIL attention and soft-target foveation. We also report kappa against consensus and expected calibration error.

As a transfer test, we freeze $g_s$ trained on Gleason2019 and apply it to ISUP grading slides. We ask whether slide-level intrinsic entropy tracks 20-pathologist disagreement, with the 0.728 from the Latent Doctor Model as the reference.

A negative result on the third check with a positive result on the first would still be useful. It would show that resolution-driven and intrinsic disagreement are distinct, even if they are not worth separating for zoom allocation on this benchmark.