Where Do Pathologists Disagree? Learning Input Regions of Rater-Specific Bias

When eleven expert pathologists grade the same biopsy, they often disagree. Current models of this disagreement give each expert one fixed personality: a dedicated network trained on that expert's labels, or a single point in a learned "style" space. These descriptions apply equally to every slide the expert reads. Clinically, that seems unlikely. A pathologist may be conservative on inflamed tissue and liberal on regenerative tissue. The same two experts may agree on clear high-grade dysplasia and part ways only on one narrow kind of borderline morphology.

This proposal asks whether expert bias depends on the input in a piecewise way, and builds a model that tests this directly. The model divides the space of slides into a small number of regions. Within each region, each rater has their own consistent pattern of errors relative to a hidden true grade, and the boundaries between regions are chosen so that those patterns are as uniform as possible inside each one. If such regions exist, three things follow from one compact model. First, it should predict what an individual expert would say more accurately than any model that treats that expert's bias as global. Second, it should run as a single network rather than a panel of eleven, which matters for gigapixel slides and 3D volumes. Third, it should produce a map of where diagnostic subjectivity actually lives, including the "indefinite for dysplasia" cases that current work discards.

Core Claim

Our hypothesis is that rater bias is input-dependent and concentrated in a few identifiable morphological regimes. We further hypothesize that partitioning the input space by rater-disagreement structure, rather than by visual similarity, yields better per-rater prediction, calibration matching the panel's, and single-model inference. The claim can fail cleanly. If one global error pattern per rater explains held-out labels as well as region-specific patterns do, the hypothesis is wrong. In that case, the existing global-rater models are already the right description.

Two lines of prior work motivate this, and each leaves the gap open from a different side. The first is input-space partitioning. Mitigating Representation Bottlenecks in Multiple Instance Learning clusters slide embeddings with K-means, trains one expert model per cluster, and distills them into a single student. It reports accuracy gains across six slide-classification architectures, largest on BRACS. Its partitions, however, are purely morphological, and it trains only on single consensus labels. Hierarchical Divide-and-Conquer Grouping for Classification Adaptation of Pre-Trained Models recursively splits the label space so that confusable classes are handled separately. It shows that isolating ambiguous groups reduces confusion, but it works on natural images with no notion of human raters.

The second line models raters globally. More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel trains one attention-based multiple-instance model per pathologist on Barrett's esophagus slides. A "multiple-instance" model treats a slide as a bag of patches with one slide-level label. The full panel of these models beats consensus training, but every virtual rater sees the whole, unpartitioned slide space. The paper also drops the 145 indefinite cases. Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation (Pionono) gives each rater one Gaussian in an 8-dimensional latent space, with the same mean and spread for every image. The Latent Doctor Model for Modeling Inter-Observer Variability learns a doctor embedding from a support set of labeled examples. Its conditional variant makes the latent prior depend on the image, which captures how ambiguous a case is overall, but each doctor still has a single global embedding.

The proposal reuses the partition-then-compress logic of the first line. It replaces the line's notion of a good partition with one defined by the second line's object of study.

The Model

Each slide $x$ passes through a frozen pathology foundation encoder and an attention-based pooling layer to produce an embedding. A small gating network assigns $x$ soft membership $g_k(x)$ in $K$ regions. A shared classifier gives a distribution $p(y \mid x)$ over the latent true grade $y$. Each rater $r$ has a confusion matrix $C_{r,k}$ in each region $k$, giving the probability that rater $r$ reports label $l$ when the true grade is $y$. The probability that rater $r$ reports $l$ is

$$ p(l \mid x, r) = \sum_{k=1}^{K} g_k(x) \sum_{y} p(y \mid x)\, C_{r,k}[y, l]. $$

Training maximizes the log-likelihood of all observed (slide, rater, label) triples. Missing rater-slide pairs are simply skipped, so incomplete panels pose no problem.

"Indefinite for dysplasia" enters as an allowed reported label $l$ but not as a latent grade $y$. A region where some raters route many latent low-grade cases to indefinite then becomes an explicit, inspectable regime of ambiguity, rather than a set of discarded cases. To keep the model identifiable and prevent region-level overfitting, each $C_{r,k}$ is shrunk toward the rater's global matrix $C_r$ with a learned strength. With $K=1$, the model reduces to a global per-rater error model, which is the hypothesis we want to reject.

We choose $K$ by held-out rater-label likelihood. This gives a principled answer to the cluster-number problem that both partitioning papers leave open. The gating network learns region boundaries end to end, so the boundaries follow wherever rater error patterns change, not wherever the slides happen to look different. At inference the model is one forward pass: one encoder, one pooling layer, a gate, and $K$ times $R$ small matrices, where $R$ is the number of raters. It outputs the panel's predicted label distribution, each rater's predicted label, and the region assignment.

Research Program

The work proceeds in three parts. The first is a measurement study, which is worth publishing even if the model wins by little. We fit the model on multi-rater pathology data at increasing $K$ and report the held-out per-rater likelihood gain over $K=1$. We then characterize the learned regions. For each region we record which morphology it captures, as read by a pathologist reviewing representative tiles, which raters change behavior there, and how much of the panel's total disagreement, measured as entropy of the panel label distribution, falls inside it.

The second part asks whether these regions are real structure or an artifact. We test whether the regions are stable across random seeds and data splits, using normalized mutual information between partitions. We also test whether they recur when the same raters grade a held-out test cohort.

The third part tests two practical consequences. The first is that a new rater can be added from a few labels. With region-specific shrinkage, we estimate their $C_{r,k}$ from a small support set and compare this against the Latent Doctor Model's support-set embedding. The second is that the single model matches the full virtual panel's calibration at a fraction of its inference cost.

The primary dataset is the Dutch Esophageal Pathology Panel used by the virtual-panel paper: 1,577 Barrett's esophagus cases with about 11 pathologists each, including the indefinite cases. Access depends on its authors, and that dependency is the main logistical risk. The public fallback and generalization sets are those in the research file. Gleason2019 has 244 prostate tissue-microarray cores and 6 pathologists. The 3k Buds dataset has 3,000 colorectal tumor-bud candidates and 9 pathologists. The ISUP dataset has 100 prostate slides and 20 pathologists. ISUP is small, but its large panel makes it the best test of distribution matching.

Baselines

The naive reference is a single attention-based multiple-instance model trained on panel majority-vote labels, with identical encoder, pooling, and tuning budget. On the Barrett's data, the virtual-panel paper reports this consensus model at 72.9% average F1 across all annotations and 0.183 expected calibration error. We will reproduce it on our splits before comparing anything else.

The strongest existing method is the Full Panel from More is more. This is one attention-based model per pathologist, with predictions ensembled across all converged raters. The paper reports 78.2% average F1 and 0.050 expected calibration error on the Barrett's test cohort, and it outperforms its "golden set" of top raters. This baseline is the strongest competitor because the QUBIQ challenge (QUBIQ) found that simple ensembles of per-rater or per-subset models beat probabilistic latent models at reproducing human uncertainty. Ensembles are therefore the ones to beat.

We will run the Full Panel with the same frozen encoder and the same per-model tuning budget as our model's total budget. We will report its inference cost in wall-clock time and memory per slide. Beating it means matching its panel-level F1 and calibration, within confidence intervals from bootstrap over test cases, while also exceeding it on held-out per-rater log-likelihood at roughly one-eleventh of its inference cost.

On Gleason2019 the distributional baseline is the Conditional Latent Doctor Model (The Latent Doctor Model). It reports a KL divergence of 0.461 to the empirical expert label distribution, and a Spearman correlation up to 0.728 between its predictive entropy and the 20 pathologists' entropy on ISUP. We will retrain it under our splits and encoder. On 3k Buds there is no reported result found on Lacuna for any multi-rater method's numerical score, so all methods there will be compared only against our own reproductions.

Two ablations isolate the key ingredient. The first sets $K=1$, giving a global per-rater confusion model, so any gain from partitioning is measured against it directly. The second fixes the partition to K-means on slide embeddings at the same $K$, in the style of the divide-and-distill paper, with per-region confusion matrices fitted on top. This separates "partitioning helps" from "partitioning by disagreement structure helps."

Why not just…?

Why not just train one network on the panel's soft label distribution? For the panel-level target, this is a real threat. A single model trained to match each slide's empirical label frequencies may reach Full Panel calibration with no partitioning at all. We include it as a baseline. If it matches our model on panel KL divergence and calibration, we will report that plainly. It cannot, however, predict what a specific pathologist would say, and it cannot absorb a new rater. Our core claim rests on held-out per-rater likelihood, where soft-label training has nothing to offer.

Why not just give a shared network a rater embedding that interacts with image features? For example, the rater embedding could modulate the image features through a learned scaling and shift. This is the most serious threat. In principle such a network can represent input-dependent bias continuously, and the conditional Latent Doctor Model and Pionono are close relatives. We will run this modulated network with matched parameters and tuning. If it matches our model on per-rater likelihood, the predictive argument for hard regions fails. What would remain is interpretability and few-label rater addition. We will test the latter directly, because a dense interaction network has many more rater-specific parameters to fit from a small support set than a shrunk confusion matrix per region. If the modulated network wins on both, we drop the model and keep only the measurement study.

Why not just distill the Full Panel into one student, as divide-and-distill does? Distillation addresses the cost problem the QUBIQ participants raised. A student matching all eleven virtual raters' outputs is a strong, cheap baseline, and we include it. But it inherits the panel's assumption that each rater's model is global, and it offers no account of where disagreement sits. Comparing it to our model on per-rater likelihood, and on the stability of the learned regions, shows whether explicit regions add anything beyond compression.

Decisive Checks

The first decisive check is the held-out per-rater log-likelihood curve as a function of $K$ on the Barrett's data and Gleason2019. A clear rise above $K=1$ that persists across seeds supports the hypothesis. A flat curve refutes it. The second is the disagreement-driven versus morphology-driven partition at matched $K$. If K-means regions perform as well, the claim reduces to the existing divide-and-distill recipe applied to rater labels.

The third is the comparison with the Full Panel on the Barrett's test cohort, using F1 and calibration against its reported 78.2% and 0.050, together with inference cost. The fourth checks whether indefinite cases concentrate in one or two learned regions, and whether including them hurts accuracy on definite cases. This answers the open question the virtual-panel paper left by excluding them.

Finally, a pathologist blinded to the model will review sample tiles from each region and judge whether the regions correspond to recognizable morphology. Without that, the "map of subjectivity" is only a statistical claim.