Laboratory medicine has a standard fix for a lab whose readings run high: treat each lab as an instrument with its own calibration curve. You keep the information that survives any monotone recalibration, meaning the relative ordering of measurements, and you fit the calibration separately with a handful of parameters. The sarcoma miRNA-pair classifier is a clean example. It scored only whether one microRNA exceeds another within the same sample, and this made its test portable across platforms without needing a reference gene (A novel serum miRNA-pair classifier for diagnosis of sarcoma).
We propose to treat pathologists the same way. Much of the disagreement between expert graders may be calibration disagreement: one pathologist's "grade 4" begins where another's "grade 3" ends, even though both rank the cases in nearly the same order. If that holds, a model should learn one shared, rater-independent severity score from all raters pooled. Each rater's quirks then become a few threshold parameters rather than a separate network. This is exactly the structure a small multi-rater cohort can afford to estimate.
Core Claim
We claim that on multi-rater histopathology grading, most disagreement between experts is explained by rater-specific cutpoints and noise levels on a shared severity axis. We further claim that a deep model built on that structure matches or beats free-form models of annotator disagreement at predicting the panel's label distribution. The advantage should widen as the number of training cases shrinks. Concretely, a backbone produces a scalar severity $s_\theta(x)$ for case $x$, and rater $r$ reports a grade at or below $k$ with probability
$$ P(y_r \le k \mid x) = \sigma\!\left(\frac{\tau_{r,k} - s_\theta(x)}{\beta_r}\right), $$
where $\sigma$ is the logistic function, $\tau_{r,1} < \dots < \tau_{r,K-1}$ are rater $r$'s cutpoints between the $K$ ordered grades, and $\beta_r$ is that rater's noise scale. This is the graded-response structure long used in psychometrics, here placed on top of a learned image or molecular encoder. We claim no novelty for the likelihood.
The new parts are three. First, we use the structure to measure how much of the observed disagreement is threshold disagreement. Second, we exploit it for sample efficiency: every rating of every case updates one shared severity function, and a new rater is calibrated from roughly $K$ parameters. Third, we apply the same "compare within source, calibrate per source" principle to the molecular features themselves. The result is a single mechanism for standardizing both the measurement and the label.
To represent cases that are intrinsically ambiguous, the severity can be made uncertain per case, with $s_\theta(x) \sim \mathcal{N}(\mu_\theta(x), v_\theta(x))$. A large $v_\theta(x)$ then spreads the panel across grades regardless of rater thresholds. Separating case ambiguity $v_\theta$ from rater calibration $(\tau_r, \beta_r)$ is the decomposition the field currently lacks.
Position Relative to Prior Work
Two multi-rater methods motivate this work and supply its strongest competitors. The Latent Doctor Model captures each doctor's style as a continuous latent vector inside a conditional variational autoencoder. It infers that vector from a support set of that doctor's cases, and in its conditional variant it lets the image control how ambiguous a case is. The virtual expert panel trains one attention-based multiple-instance model per pathologist on Barrett's esophagus slides and averages them. This improved F1 and cut calibration error sharply relative to consensus training.
Both papers show that preserving disagreement pays off. Both give the rater an unconstrained representation: a free latent in one case, an entire network in the other. We impose the opposite: the rater is permitted to shift and stretch the severity axis but not to reorder cases. If that restriction is roughly true, it should be far more data-efficient. If it is false, our decomposition will reveal that directly.
The virtual-panel paper also removed the "indefinite for dysplasia" category. An ordinal model can instead place that category between non-dysplastic and low-grade and estimate each pathologist's threshold for using it.
Ranking-based grading is the closest methodological ancestor. Huang et al. turned cataract grading into learning a severity order with a smooth ranking loss, specifically to escape observer disagreement. However, they trained against single consensus grades and never modeled individual raters. Our model keeps the continuous severity idea and adds explicit per-rater calibration.
Soft-label training on merged rater maps, as in Gleason Score Prediction using Deep Learning in Tissue Microarray Image, removes rater identity entirely. Diagnosis-first weighting of raters (DiFF) needs segmentation masks and weights raters by their usefulness for a downstream task. It does not address grade thresholds, so it plays only a background role here.
Three results constrain the design. First, a Transformer trained on about 500 TCGA prostate cases converged unstably, and random forests failed on Gleason 8 (Discovering Multi-omic Biomarkers for Prostate Cancer Severity). For that reason, the molecular encoder stays shallow. Second, fixation chemistry can destroy signal that no normalization recovers (Xu et al.), so we do not claim that within-sample ratios repair degraded samples. Third, the binary pair classifier's sensitivity fell from 100% to 50.0% on 35 external samples. We read that as a warning that throwing away the size of a ratio is costly, which is why our molecular features keep continuous log-ratios rather than 0/1 inversions.
Research Program
The first component is a measurement study that stands independently of any new model. On each multi-rater dataset we fit, directly to the rating matrix, a nested series of models. The first shares one set of cutpoints across all raters with no image model. The second gives each rater its own cutpoints. The third also gives each rater its own noise scale. The fourth adds per-case ambiguity.
The gain in log-likelihood at each step shows how much disagreement comes from thresholds, from rater reliability, and from the cases themselves. The residual that remains is ordering disagreement, meaning pairs of cases that two raters rank in opposite orders even after calibration. We will also report a nonparametric check alongside the fitted models: pairwise rank concordance between raters, compared with their exact-grade agreement. A large gap between the two supports the premise.
The second component is the model itself. We attach the graded-response head to standard pathology backbones: attention-based multiple-instance pooling for whole slides and a patch encoder for tissue microarray cores. All raters' labels are trained jointly under the likelihood above. We will also train a variant that uses only pairwise orderings within each rater, which is the direct analogue of within-sample miRNA pairs. This checks whether the cutpoint parameterization or the invariance itself is doing the work.
Adapting to a new rater means freezing the backbone and fitting $\tau_r, \beta_r$ from a small support set. This mirrors the Latent Doctor Model's support-set protocol.
The third component applies the principle to molecular data in TCGA-PRAD. Gleason grades there come from pathologists at many tissue source sites, and RNA profiles carry site batch effects. We treat each site as a rater and give it its own cutpoints. We represent expression as continuous within-sample log-ratios over a curated gene set. This removes per-sample scaling in the way the sarcoma pairs did, but keeps the size of each ratio.
A shallow encoder, either penalized linear or a small multilayer perceptron, maps these ratios to severity. The question is whether site-specific calibration of the labels, combined with site-invariant features, improves grading on held-out sites, especially the Gleason 8 transition cases. Site calibration is confounded with site case mix. The within-sample features are what make the cutpoints identifiable, and we will report the cases where they are not.
Baselines
The naive reference is a backbone trained with cross-entropy on majority-vote labels, with standard augmentation, evaluated as a panel predictor through its softmax. Because the consensus model in the Barrett's study reached an expected calibration error of 0.183, we expect it to be poorly calibrated. It remains the level any method must clear.
The second baseline is soft-label training on the rater-averaged distribution, as in the Gleason2019 U-Net study (Gleason Score Prediction). That study reports a combined challenge score of 0.778, but for segmentation rather than panel-distribution prediction, so we use it as a method rather than as a number to beat.
The strongest competitor for distribution prediction on Gleason2019 and ISUP is the Latent Doctor Model. It reports a Kullback–Leibler divergence of 0.461 to the expert distribution on Gleason2019, and a Spearman correlation of up to 0.728 between predicted entropy and the 20-pathologist entropy on ISUP. For 3k Buds, no reported result was found on Lacuna for the Latent Doctor Model beyond "competitive with deep ensembles."
On the Barrett's panel, the strongest competitor is the virtual expert panel, with full-panel F1 of 78.2% and expected calibration error of 0.050, with indefinite cases excluded.
For fairness, we will reimplement both competitors on the same backbones, the same folds, and the same case subsamples, giving every method the same number of hyperparameter trials and GPU hours. We will also report our numbers under their published protocols where feasible. Beating a competitor means lower KL divergence and higher entropy correlation at full data with overlapping confidence intervals excluded, or a statistically significant gap at 25% and 50% of training cases. Matching them at full data while winning at small data would still support the claim.
On TCGA-PRAD, the reference is the random forest of Discovering Multi-omic Biomarkers. It reports median F1 of 0.70 for Gleason 7 from RNA and 0.75 from RNA plus copy number. It gives no usable Gleason 8 number, only that models "consistently struggled." We will rerun it on site-held-out splits, because the published figures are not site-held-out.
Two ablations isolate the key ingredient. The first is a per-rater head model, a separate categorical output per rater on a shared backbone. It has the same rater identity information but no shared ordering constraint. If it matches our model, the monotone structure is not what helps. The second is a model with shared cutpoints across raters. If it matches, rater calibration is not what helps.
Why not just…?
Why not just train on the majority vote with heavy augmentation? The consensus label discards exactly the quantity we are trying to predict. Existing evidence already shows the cost: in the Barrett's study, consensus training gave calibration error of 0.183 against 0.050 for a rater-aware panel (virtual expert panel). We keep this baseline in every table so that readers can see the size of the gap on each dataset.
Why not just use the virtual expert panel or the Latent Doctor Model, which already model raters? This is the real threat. A per-rater network can learn anything our cutpoints can. A free latent vector can in principle encode a threshold shift. The case for our model is statistical. The virtual panel trains each rater on under 200 of that rater's own cases, whereas our severity function sees every rating of every case. We therefore predict our advantage will grow as cases are removed and will be largest for rarely used grades such as Gleason 5 and indefinite-for-dysplasia. If the subsampling curves show no widening gap, we will conclude that free-form rater models are sufficient, and the measurement study remains the main contribution.
Why not just fit an ordinal regression with rater offsets, a well-known statistical model? Our model essentially is that. We say so openly rather than claiming a new likelihood. What is missing from the multi-rater deep learning work in the file is fitting this structure on a learned encoder, comparing it head to head with free-form rater models, and using the fitted decomposition to test whether disagreement is really about thresholds. The pairwise-only variant tests whether the invariance or the particular link function carries the gain.
Why not just skip the labels and predict grade from molecular biomarkers, which are objective? On TCGA-PRAD, molecular random forests reach only 0.70 to 0.80 median F1 and fail on Gleason 8 (Discovering Multi-omic Biomarkers). They are also trained on the same subjective grades, now from many sites with different calibrations. Our molecular arm keeps the molecular features but corrects the labels' site calibration. That is a direct test of whether some of the "molecular ceiling" is actually label calibration noise.
Proposed Experiments and Decisive Checks
The first decisive check is the decomposition on Gleason2019 (244 cores, 6 pathologists) and ISUP (100 slides, 20 pathologists). If per-rater cutpoints and noise scales explain less than half of the likelihood gain between a no-disagreement model and the full per-case model, the premise is weak. In that case we would report the result as a negative finding about the nature of rater disagreement and not proceed to strong modeling claims.
The second check is the subsampling curve. We train on 25%, 50%, and 100% of cases, with five repeats each. We report KL divergence to held-out panel distributions, entropy rank correlation, expected calibration error, and per-grade recall for the highest grades. Where the dataset licensing allows, we run on the Barrett's panel with indefinite cases included, reporting both with and without them for comparison against the published setting.
The third check is new-rater adaptation. We hold out one rater at a time and calibrate from 10, 20, and 40 of their cases. We then compare predicted and actual ratings on their remaining cases against the Latent Doctor Model given the same support sets.
The fourth check is the TCGA-PRAD site-held-out experiment. We measure macro F1 and Gleason 8 recall for continuous log-ratio features with site cutpoints, against z-scored features, binary pair inversions, and the random-forest reference.
The proposal succeeds if the decomposition shows threshold disagreement dominating and the structured model wins specifically in the small-data and rare-grade regimes. It fails cleanly if the per-rater head ablation matches it everywhere.
References
- A novel serum miRNA-pair classifier for diagnosis of sarcoma
- The Latent Doctor Model for Modeling Inter-Observer Variability
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- A Computer-Aided Diagnosis System of Nuclear Cataract via Ranking
- Gleason Score Prediction using Deep Learning in Tissue Microarray Image
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Discovering Multi-omic Biomarkers for Prostate Cancer Severity Using Machine Learning
- Crosslinking intensity modulates the reliability and sensitivity of chromatin conformation detection at different structural levels