When eleven pathologists look at the same Barrett's esophagus biopsy, they often disagree, and the disagreement is not random. Some cases are hard because the tissue really sits on the boundary between two grades. Some pathologists are systematically stricter than others. Some calls are inconsistent from one case to the next. Educational testing has a standard tool for separating these three things: the Rasch model, a latent-trait model that explains each answer through two quantities, how able the test-taker is and how hard the question is. We propose bringing that structure into deep learning for multi-rater grading. A neural network reads the slide and places it at a point on a continuous severity scale, together with a spread that measures how ambiguous it is. Each pathologist is described by a small set of numbers: a severity shift that moves their grade boundaries, and a consistency score.
The bet is that this explicit split does three things current multi-rater models cannot do together. It predicts how a specific named pathologist would grade an unseen slide. It calibrates a new pathologist from a handful of labels. It keeps the "indefinite for dysplasia" cases that existing pipelines throw away, because on a continuous scale those cases are just an interval near a boundary. If a simpler two-stage recipe, or plain soft-label training, matches it on these tasks, the proposal fails, and the experiments are built to show that.
Core Claim
Diagnostic grades such as Barrett's NDBE/IND/LGD/HGD or ISUP grade groups are ordinal: they are ordered judgements along an underlying severity continuum. Disagreement among raters can therefore be decomposed into an image-dependent case location, an image-dependent case ambiguity, and rater-specific threshold shifts and consistencies. Let $y_{ij}$ be the grade rater $j$ gives case $i$, let $x_i$ be the slide, and let $k$ index the ordered grade boundaries with shared cutpoints $\tau_k$. Then we model
$$ P(y_{ij} \le k \mid x_i) = \sigma\!\left( a_j \, \frac{\tau_k + \delta_j - \mu(x_i)}{s(x_i)} \right). $$
Here $\sigma$ is the logistic function. $\mu(x_i)$ and $s(x_i) > 0$ are the case location and ambiguity produced by a shared slide encoder. $\delta_j$ is rater $j$'s severity, where a positive value means they need more evidence before calling a higher grade. $a_j > 0$ is rater $j$'s consistency. This is the many-facet ordinal extension of the Rasch model. What is new is that the "item difficulty" side is produced from pixels rather than estimated as a free parameter per item. That is what lets the model generalize to slides no one has graded.
This differs from the closest existing work in a specific way. The Latent Doctor Model captures rater style with an unstructured Gaussian latent vector inside a conditional variational autoencoder (a generative network that samples labels through a learned latent code). It treats grades as unordered classes, and its latent code cannot be read as "stricter" or "more ambiguous." Our parameters are scalars with a fixed meaning and can be checked for the invariance property that psychometrics demands.
Prior Work and Its Role
Three lines of multi-rater work motivate the gap. Each models disagreement without an interpretable ordinal scale. The Latent Doctor Model shows that disagreement is learnable signal: its image-conditioned variant reaches KL divergence 0.461 to the expert distribution on Gleason2019, and its predicted entropy correlates with the entropy of 20 pathologists on ISUP at Spearman up to 0.728. More is more: virtual expert panel trains one attention-based multiple-instance network per pathologist on the DEPP Barrett's cohort and averages them. It improves F1 from 72.9% to 78.2% and expected calibration error from 0.183 to 0.050 over consensus training, but it excludes all IND cases (1,577 down to 1,432) and needs one network per rater. CrowdGleason learns a latent true grade jointly with per-resident confusion matrices, which are tables of how often each rater mistakes one grade for another. It reaches quadratic kappa 0.7048 versus 0.6576 for majority vote, and 0.7276 with 10% expert anchors. Its authors note that these confusion matrices ignore image features, so they cannot represent a resident who grades well-formed glands correctly but fails on cribriform patterns.
An Estimation and Analysis Framework for the Rasch Model supplies method. It gives a fast closed-form estimator for Rasch abilities that runs nearly seven times faster than MCMC on MovieLens 100k, and its error bounds stay tight with as few as 20 users. We use its estimator to initialize new-rater parameters. One limitation constrains our design: that paper handles only binary responses without image conditioning. We therefore expect to fit the ordinal and image-conditioned parts by gradient-based maximum likelihood, and we use the closed form only to start new raters cheaply.
A contrasting line removes humans from the loop instead of modeling them. The pancreatic cyst CT system in Deep learning CT model for pancreatic cystic neoplasms reports reader agreement on solid nodules as low as 62% and replaces visual cues with 63 quantitative features. It never calibrates its eight readers, even though their adoption of AI advice (87.14%) presumably depends on their individual thresholds. Our rater parameters are the missing measurement for that kind of study. We do not test this application here.
Research Program
We build one shared slide encoder: an attention-based multiple-instance network over tissue tiles, the same family the virtual-panel paper uses. It has two output heads, $\mu$ and $\log s$. A table of per-rater scalars $(\delta_j, \log a_j)$ and shared cutpoints $\tau_k$ completes the model. Training maximizes the likelihood above over all observed (case, rater, grade) triples. This handles sparse and unbalanced rating matrices directly, because missing pairs simply do not contribute. To make the scale identifiable, we fix mean rater severity at zero and mean log-consistency at zero.
The IND category takes its natural ordinal place between NDBE and LGD. A case is then "indefinite" when its location falls near the cutpoints that bound IND, or when its ambiguity is large. Either way, the model can say which.
We also study one extension in response to the CrowdGleason limitation. Rater severity is allowed to depend on the case through a rank-one term, $\delta_j + u_j^\top g(x_i)$, where $g$ is a low-dimensional projection of the encoder features and $u_j$ is a per-rater weight vector. We add this term only if the invariance test described below shows that plain scalar severities fail. That ordering keeps the interpretable model as the default, which matters because Rasch measurement is valued precisely for its invariance.
Baselines
The naive reference point is a network trained with cross-entropy on majority-vote labels, using the same encoder, the same tiles, and the same tuning budget. Its numbers in the file are kappa 0.6576 on CrowdGleason and, for consensus training on DEPP, F1 72.9% and calibration error 0.183. Our model should beat it clearly on every distributional metric. If it does not, the evaluation is broken.
The strongest existing distributional method is the image-conditioned Latent Doctor Model, whose best reported result is KL 0.461 on Gleason2019 with Spearman up to 0.728 on ISUP. We will reimplement it on the same encoder and splits. When a rater is unseen, it receives exactly the same support labels our model receives, and both get the same hyperparameter search budget. On DEPP the strongest rater-specific method is the Full virtual panel (F1 78.2%, calibration error 0.050). We rerun it both on its original IND-excluded split and on the full cohort with IND included. On CrowdGleason and SICAPv2 the reference is SVGPCR and SVGPMIX, at kappa 0.7048, 0.7276, and 0.7814 respectively, run with the same 10% expert anchors where applicable.
Beating these baselines means three things. On Gleason2019 and ISUP, we should match or lower the KL of the Latent Doctor Model while using about one thousand times fewer rater-specific parameters. On DEPP with IND included, we should match the Full Panel's calibration error. We should also lower held-out per-rater negative log-likelihood. Matching rather than clearly exceeding these baselines on aggregate metrics is acceptable, provided we win clearly on the new-rater and IND tests below, which those methods cannot address directly.
Two ablations isolate the key ingredient. The first removes the ambiguity head ($s \equiv 1$), so that location alone must explain all disagreement. The second replaces the ordinal link with unordered per-rater logit offsets, which keeps rater parameters but discards the severity scale.
Why not just…?
Why not just train on the panel's soft label distribution? Cross-entropy against each case's empirical grade histogram is cheap and may match KL to the panel as well as our model does. We consider this a genuine threat on the distribution-matching metric. It cannot say what a particular rater would call, however, and it has no way to absorb a new rater. We include it as a baseline. If it ties us on KL and on the ambiguity-entropy correlation, we will report that the gain lies only in rater-level prediction and new-rater calibration, not in panel modeling.
Why not just fit a classical many-facet Rasch model to the label matrix, then regress its case parameters from images? This two-stage recipe is the most serious alternative, because it uses the same psychometric structure with standard software. Its weakness is that case parameters estimated from three to eleven labels are noisy, and the second-stage regression treats them as exact. We run it as an explicit baseline. Joint training should help most where cases have few ratings: CrowdGleason, and DEPP after subsampling raters per case. If the two-stage recipe matches the joint model even there, we will conclude that the deep joint model is unnecessary and recommend the two-stage recipe.
Why not just keep one network per pathologist, as in the virtual panel? It works well on DEPP. But it scales linearly in networks with the number of raters, and some rater models failed to converge in the virtual-panel study. It also cannot represent a pathologist who has contributed 20 labels. Our new-rater experiment measures that gap directly.
Why not just use Dawid–Skene confusion matrices? Confusion matrices, as in CrowdGleason, have $K^2$ free entries per rater, where $K$ is the number of grades, and they ignore order. With few labels per rater they should be less data-efficient than two scalars. This is a prediction, and the label-budget curves below test it.
Proposed Experiments
The first decisive check is new-rater calibration. On ISUP (20 raters) and DEPP (about 11 raters), we hold out each rater in turn. We give the model $k \in \{5, 10, 20, 50\}$ of that rater's labels, estimate $\delta_j$ and $a_j$ starting from the closed-form initialization, and predict the rater's remaining grades. We report negative log-likelihood and quadratic kappa against the rater's own calls, compared with the Latent Doctor Model given the same support set, a confusion-matrix rater, and a population model with no rater adaptation. Success means a clear likelihood advantage at $k \le 20$.
The second check is IND retention on DEPP. We train with and without IND cases and evaluate on held-out cases. We measure whether the predicted IND probability matches the fraction of the panel who called IND. We also measure whether including IND improves or degrades calibration error and F1 on the LGD and HGD calls relative to the Full Panel trained without IND. A negative result here would be informative: it would mean IND is not a severity interval and may instead reflect technical slide quality.
The third check is psychometric invariance, the property that justifies the Rasch structure. We split cases by predicted ambiguity into easy and hard halves, re-estimate rater severities on each half, and correlate the two estimates across raters. We repeat with a random split to measure test-retest reliability. High correlation in both supports scalar severities. A low easy/hard correlation alongside a high random-split correlation is direct evidence for the case-dependent rater skill that the CrowdGleason authors suspected, and it would trigger the rank-one extension. We then evaluate that extension on cribriform-rich versus well-formed-gland subsets of CrowdGleason.
Across Gleason2019, ISUP, CrowdGleason with external SICAPv2, and DEPP, we additionally report the following measures under identical splits: KL divergence to the panel, Spearman correlation between predicted ambiguity $s(x_i)$ and panel entropy, calibration error, and kappa against expert truth where it exists. The 3k Buds categories may not be ordinal. We use that dataset as a negative control, where we expect the ordinal model to show no advantage over unordered rater offsets.
References
- The Latent Doctor Model for Modeling Inter-Observer Variability
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- The CrowdGleason dataset: Learning the Gleason grade from crowds and experts
- An Estimation and Analysis Framework for the Rasch Model
- Deep learning CT model for stratified diagnosis of pancreatic cystic neoplasms: multicenter development, validation, and real-world clinical impact