Pathologists as Weighting Functions: Separating Threshold Bias, Criterion Bias, and True Ambiguity in Multi-Rater Grading

When several pathologists grade the same prostate biopsy and disagree, current machine learning pipelines either vote the disagreement away or keep it as a soft label. Both treat each pathologist as an opaque source of labels. Decision science offers a more specific picture. Risk analysts who fill in weighting schemes such as the Analytic Hierarchy Process (a method where experts compare criteria in pairs and the comparisons are turned into numeric weights) show systematic "subjective weighting bias": each expert reads the same evidence but applies personal weights and personal cutoffs. We propose to model pathologists the same way. Each rater's grade is an ordinal decision, meaning a choice among ordered categories. That decision is computed from a shared set of measurable morphological concepts, such as cribriform area or poorly formed gland fraction, and passed through that rater's own concept weights and grade cutoffs. What remains after the weights and cutoffs are accounted for is genuine case ambiguity.

The payoff is that disagreement can be decomposed case by case and rater by rater. Some disagreement comes from a rater placing the grade boundary higher or lower (threshold bias). Some comes from a rater weighting one pattern more than peers do (criterion bias). The rest is residual ambiguity that no correction should remove. The first two can be corrected before training, which targets the bias amplification that Lochau et al. documented. The third should be preserved, as Schoppe et al. argue.

Core Claim

A structured rater model, built from shared concept scores, per-rater concept weights, per-rater ordinal cutoffs, and a small residual term, will explain inter-observer disagreement about as well as an unstructured latent-variable model while making its sources explicit. It also allows a debiased consensus. From a panel of only three pathologists, that consensus should agree better with a large held-out panel than a three-rater majority vote does. Models trained on it should carry less identifiable rater signature than models trained on any single rater.

Concretely, rater $r$ assigns case $x$ to grade $k$ when

$$ \tau_{r,k-1} < w_r^\top c(x) + \epsilon_{r}(x) \le \tau_{r,k}, $$

where $c(x)$ is a vector of concept scores from a frozen concept extractor, $w_r$ is rater $r$'s concept weighting, $\tau_{r,k}$ are that rater's cutoffs between grades, and $\epsilon_r(x)$ is a noise term whose spread is shared across raters but varies by case. The weights and cutoffs are fit hierarchically, so each rater's parameters are shrunk toward a panel mean $(\bar w, \bar\tau)$. The debiased label for a case is its grade distribution under $(\bar w, \bar\tau)$, with the case-level spread of $\epsilon$ kept as ambiguity.

What This Builds On and Where It Differs

The motivating evidence comes from both directions. In risk assessment, Nie et al. show that a self-consistent expert can still be systematically skewed, and Đjekić et al. show that consensus panels can amplify a shared bias rather than cancel it. In pathology, discordance reaches 50% in Gleason grading (Koziarski et al.). Lochau et al. found that a classifier could identify the source rater more easily from network predictions than from the raw annotations: accuracy rose from 0.87–0.917 to 0.90–0.93. Networks therefore amplify rater style instead of averaging it out.

The closest existing method is the Latent Doctor Model of Schoppe et al.. It encodes each rater from a support set of their past diagnoses and samples a latent vector to simulate a panel. It matches panel entropy well, but its latent vector has no meaning, so it cannot say whether a rater differs in threshold or in criterion weighting, and it offers no debiased consensus. Our model trades flexibility for structure. Whether that trade costs accuracy is an empirical question the experiments settle.

GleasonXAI supplies the method we reuse. It segments guideline-defined patterns and is trained on STAPLE-aggregated soft labels from 54 pathologists. It then maps concepts to grades with fixed guideline rules, whereas we learn how each rater actually maps concepts to grades.

Wilm et al. constrain the design. On canine mitoses, unweighted majority consensus saturates after about three annotators, and training on the three least agreeable annotators performed about as well as training on the three most agreeable (F1 0.756 vs 0.751). This warns that "reliability weighting" of whole annotators may buy little. We therefore correct how raters map evidence to labels instead of down-weighting raters wholesale.

Finally, we deliberately do not transplant the Entropy Weight Method from Wang or its test-time use in Cyclic TTA onto annotators. That method gives more weight to whatever varies more across cases. A rater who labels at random has high dispersion and would be rewarded.

Research Program

The first step is building the concept extractor. We train a GleasonXAI-style segmentation network on the 1,015-image multi-expert prostate tissue microarray cohort and freeze it. For each slide or core it produces $c(x)$: area fractions and counts for each guideline pattern, plus a detection flag for rare patterns such as comedonecrosis. Because rare concepts are exactly where Kather et al. report failures, we record extractor recall per concept. Rater weights on poorly detected concepts are then reported as unidentifiable rather than interpreted.

The second step is fitting the rater models on the ISUP grading dataset (100 slides, 20 pathologists) and on Gleason2019 (244 cores, 6 pathologists). The model is a hierarchical Bayesian ordinal regression with per-rater weights and cutoffs. We compare nested variants on held-out labels: cutoffs only, cutoffs plus weights, and each of these with or without case-dependent noise. This nesting is the decomposition: the likelihood gained by each added component measures how much of the disagreement that source explains.

The third step is ground-truth validation. MCDM work has no ground truth, but pathology panels partly do. We use three substitutes for ground truth. The first is synthetic raters: we perturb a reference rater's weights and cutoffs by known amounts and check that the fit recovers them. The second is a large held-out panel, where a debiased consensus from a small rater subset is scored against the grade distribution of the remaining raters. The third is the identifiability test of Lochau et al., applied to classifiers trained on debiased versus per-rater labels.

Baselines

The naive reference point is hard majority voting. On a given rater subset, it produces a single label per case, which is used both as the consensus estimate and as the training target. Any claim of debiasing must first beat this with the same raters.

Soft-label averaging is the empirical distribution of the subset's grades. It is what GleasonXAI trains on in STAPLE form and reports a Dice of 0.713 versus 0.691 for hard labels (Kather et al.). It keeps ambiguity but also keeps rater bias.

A per-rater confusion-matrix model, meaning a Dawid–Skene-style or STAPLE-style aggregator, is the strongest classical competitor for removing threshold bias. It is fit on the same labels without image features. We found no reported result on Lacuna for such an aggregator on the ISUP or Gleason2019 panels, so we will run it ourselves.

The strongest learned baseline is the Latent Doctor Model and its conditional variant from Schoppe et al.. They report a KL divergence of 0.461 to the empirical label distribution on Gleason2019 (conditional model) and a Spearman correlation of up to 0.728 between model entropy and 20-pathologist panel entropy on ISUP. We will reimplement it with the same splits, the same support-set sizes, and a backbone matched to our concept extractor, and give it an equal hyperparameter search budget. Beating it means lower KL on held-out rater distributions and higher entropy correlation. Tying it within confidence intervals while adding interpretable decomposition would also count as a positive result. Losing clearly would mean the structural constraint is too restrictive.

Two ablations isolate the key ingredient. One fixes all weights to the panel mean and allows only per-rater cutoffs, which tests whether criterion weighting matters beyond threshold shift. The other replaces the frozen concepts with a generic slide embedding of the same dimension, which tests whether the concept basis is what makes the decomposition meaningful.

Why not just…?

Why not just use a confusion-matrix aggregator like Dawid–Skene or STAPLE? This is the most serious threat. For ordinal grades, a per-rater confusion matrix already absorbs much of the threshold shift. What it cannot represent is that a rater's errors depend on the image, for example over-grading only when cribriform glands are present. The cutoffs-only ablation against the full model settles this directly. If adding weights yields no held-out likelihood gain and no better agreement with the large panel, criterion bias is negligible on these data and the confusion-matrix aggregator should be preferred. We would report that outcome.

Why not just use the Latent Doctor Model, which already conditions on rater identity? It likely matches or exceeds us in fit, since its latent vector is unconstrained. However, an unconstrained latent cannot tell a clinician that rater 7 places the grade 2/3 boundary higher, and it provides no recipe for a debiased target. The head-to-head comparison above tells us how much fit, if any, the structure costs.

Why not just weight annotators by entropy or agreement, as in hybrid MCDM? Entropy weighting rewards dispersion regardless of correctness. Agreement weighting is undercut by Wilm et al., where agreeable and disagreeable annotator trios trained equally good detectors. Agreement weighting also inherits the shared-bias problem of consensus panels that Đjekić et al. describe. We include an entropy-weighted consensus and an agreement-weighted consensus as extra rows in the held-out-panel test so this is checked rather than asserted.

Why not just skip rater labels and train weakly supervised on slide grades? Silva-Rodríguez et al. show slide-level training can generalize better than pixel-supervised training (quadratic kappa 0.83 vs 0.77). But the slide grades are themselves rater decisions, so rater bias moves into the target instead of disappearing. Our debiased labels can serve as that slide-level target. In that sense this is complementary, and we test it in the identifiability experiment.

Proposed Experiments

The decisive check is subset-to-panel agreement on ISUP. We repeatedly draw three of the 20 pathologists and form four consensus estimates from their labels: majority vote, soft average, confusion-matrix aggregation, and our debiased consensus. Each is scored by KL divergence and quadratic kappa against the distribution of the 17 held-out raters. We repeat for subset sizes one through ten to trace scaling curves comparable to Wilm's saturation result. If debiased three-rater consensus does not beat three-rater majority against the held-out panel, the core claim fails.

Second, we test prediction of new raters on both panels. Given a support set of $m$ labeled cases from an unseen pathologist, we fit only their weights and cutoffs and predict their remaining grades. We compare against the Latent Doctor Model at matched $m$, since both methods adapt from a few labels.

Third, we test rater identifiability. We train one slide classifier per rater on raw labels and one on debiased labels, then train a discriminator to identify the source rater from each classifier's predictions, following Lochau et al.. Success means identifiability of debiased-trained models falls toward chance, while agreement with the held-out panel does not drop.

Fourth, we run synthetic recovery. Known perturbations of weights and cutoffs are injected into a reference rater, and we report recovery error as a function of the number of cases and raters. This establishes whether 100 slides can support the model at all.

As a secondary, simpler regime, we apply a cutoffs-only version to the 12-rater canine mitosis cohort of Wilm et al.. There, $c(x)$ is a single frozen detector score per candidate, and we test whether per-rater cutoff correction breaks the 0.74–0.76 F1 ceiling against a leave-annotators-out consensus.

A limitation is inherited from Nie et al.: bias shared by the entire panel sits in $(\bar w, \bar\tau)$ and is invisible to this method. We state this plainly. Where ISUP guideline rules give a reference mapping from patterns to grades, we compare $\bar w$ against those rules, the only external anchor available.