When several pathologists grade the same tissue, they often disagree, and medical AI systems are trained on the mix of their labels. The usual learning-theory question is how much a trained model changes if one image in the training set is swapped for another. That is the wrong question here. The variation that matters comes from people: if a different pathologist had been hired, a whole block of labels would change at once. This note proposes treating the pathologist, not the image, as the unit that gets swapped. We define how much a trained model shifts when one rater's labels are replaced by another rater's, derive a simple bound on that shift, and use it to explain and replace a puzzling empirical practice. That practice is discarding discordant annotators, which improves models even though it throws away data.
The bound makes two predictions that can be checked, and each has a practical consequence. First, at a fixed labeling budget, how well a model agrees with pathologists it has never seen should depend mainly on how many raters contributed and how evenly, not on how many images were labeled. Second, a rater hurts the model roughly in proportion to their share of the labels multiplied by how often they disagree with colleagues. So capping each rater's influence during training should recover the benefit of filtering raters out, without needing a trusted "anchor" pathologist to decide who is good.
Core Claim
Algorithmic stability is the study of how much a learning algorithm's output moves when its training data is perturbed. Its classic tool is the twin-dataset experiment: train on two datasets that differ in one sample and compare the resulting models. Train faster, generalize better: Stability of stochastic gradient descent showed how to bound this movement for stochastic gradient descent (SGD). It tracks how the two training runs drift apart: the gap grows only on steps where the differing sample is drawn, and by at most an amount set by the step size. In that bound, the generalization gap scales like $L^2\sum_t \alpha_t / n$ for convex, $L$-Lipschitz losses, where $\alpha_t$ is the step size at step $t$ and $n$ the number of samples.
We apply the same argument with one change: the perturbation replaces all labels from one rater. Suppose raters $r_1,\dots,r_m$ label a pool of images. Rater $j$ supplies a fraction $\pi_j$ of all training labels. On the images rater $j$ labeled, a replacement rater drawn from the same population disagrees with $r_j$ on a fraction $\delta_j$ of them. Define rater-replacement stability $\beta_{\text{rater}}$ as the largest expected change in test loss when any single rater's labels are replaced this way. The two SGD runs only drift apart on steps that draw an image whose label actually changed, and a fraction $\pi_j\delta_j$ of steps do so. The convex analysis of Hardt et al. therefore gives
$$ \beta_{\text{rater}} \;\le\; 2L^2 \Big(\sum_t \alpha_t\Big)\, \max_j \pi_j \delta_j . $$
Here $\beta_{\text{rater}}$ in turn bounds the gap between the model's loss against its training raters and its expected loss against a fresh rater from the same population.
The derivation is a straightforward adaptation, and it inherits the worst-case pessimism and convexity assumptions of its parent. It is useful for three predictions it makes that current multi-rater work never tests. With $m$ raters sharing equally, the rater-level gap falls like $1/m$, whatever the number of images. Adding more images from the same few raters does not shrink it. Finally, the harm is concentrated in raters who have both a large share and high disagreement.
This also addresses a tension the file identifies. Variability Matters: Evaluating Inter-Rater Variability in Histopathology for Robust Cell Detection found that training cell detectors on the top 50–75% most conforming pathologists beats training on all 120. Image-level stability, which improves with every added sample, cannot explain that. Rater-level stability can, if the discarded raters are exactly those with large $\pi_j\delta_j$. The same bound implies a gentler fix: cap each rater's influence rather than delete them.
Relation to Prior Work
Three stability results supply the method. Train faster, generalize better: Stability of stochastic gradient descent provides the drift-counting argument we adapt from samples to raters. Leave One Out Error, Stability, and Generalization of Voting Combinations of Classifiers proves that bagging, meaning averaging models trained on random subsamples, improves stability. We test the rater-level analogue: averaging models trained on subsamples of raters rather than of images. Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernel shows that early-stopped gradient descent reaches the label-noise floor $\sigma^2$. It treats that noise as unstructured and independent across samples, which is exactly the assumption that rater-correlated disagreement breaks. All three perturb one sample with one label, and none considers correlated, block-structured label changes.
On the other side, several works motivate the proposal by showing that modeling raters helps, but none asks how sensitive training is to which raters were hired. The Latent Doctor Model in The Latent Doctor Model for Modeling Inter-Observer Variability learns a style embedding per doctor. Pionono in Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation models each rater as a Gaussian in a latent space. SVGPCR in The CrowdGleason dataset: Learning the Gleason grade from crowds and experts estimates rater reliability with Gaussian processes. All three evaluate against the same raters or a fixed expert reference. We add an evaluation in which entire raters are held out, which is the setting deployment actually faces.
DGN: Disagreement Graph Networks for Learning from Multiple Annotators ranks annotators without ground truth using a disagreement graph. Its disagreement estimate is close to our $\delta_j$, but it ignores each rater's share $\pi_j$ and assumes uniform class noise, which does not fit ordinal Gleason grades. We reuse its idea of anchor-free disagreement estimation and change what the estimate is used for. Variability Matters constrains the design. It is the empirical result we must reproduce and explain, and its dependence on an anchor annotator is the limitation we aim to remove.
Research Program
The first component is measurement. On each benchmark we run rater-level twin experiments. We train a model, then retrain it with one rater's labels replaced by another rater's labels on overlapping images, or with that rater's images relabeled by colleagues' majority where no overlap exists. Every other source of randomness stays fixed, including initialization and data order. We record the change in predictions on held-out images. Plotting this change against $\pi_j\delta_j$ across raters tests whether the bound's key quantity actually orders raters by their influence in deep networks, where the convex bound does not formally apply.
The second component tests the scaling prediction. The 120-pathologist cell detection benchmark is large enough to vary the number of raters $m$ and the number of labeled patches $n$ independently. We fix $n$ and spread it across $m \in \{5, 10, 20, 40, 80\}$ raters. Separately, we fix $m$ and vary $n$. We measure performance against a disjoint panel of held-out raters. The prediction is that error to held-out raters falls with $m$ at fixed $n$, and levels off with $n$ at fixed $m$. That result would directly answer how to split an annotation budget between more pathologists and more slides per pathologist.
The third component is the intervention, rater-capped sampling. We estimate $\delta_j$ from each rater's disagreement with others on shared images, which needs no anchor. We then set each rater's sampling probability so that $\pi_j\delta_j$ never exceeds a cap $c$, and tune $c$ on held-out raters. Filtering is the special case where a rater's weight is set to zero. Rater bagging, which averages models trained on random subsets of raters, is a second intervention motivated by the voting-ensemble result.
Baselines
The naive reference points are pooled training and majority vote. Pooled training uses every rater's label as a separate example with no rater handling. Majority vote trains on the consensus label. On CrowdGleason, The CrowdGleason dataset reports majority vote at quadratic Cohen's kappa 0.6576. These show whether any rater-aware method is needed at all.
For cell detection, the strongest competitor is conformity filtering from Variability Matters. It trains DeepLabv3+ on the top quartiles of raters ranked by modified F1 against an anchor annotator. It is the only method shown to beat full-data training on the 120-pathologist benchmark, and it is exactly the effect we claim to explain. The file reports only relative results for it; there is no reported raw F1 on Lacuna for the P50–100 subset. We therefore rerun it ourselves with the same DeepLabv3+ backbone, patches, and training schedule.
For Gleason patch grading, the strongest reported baseline is SVGPCR at quadratic kappa 0.7048 on CrowdGleason. SVGPMIX reaches 0.7276, but it uses 10% expert labels, so we compare against it only in a matched-supervision setting. For segmentation on Gleason 2019, the strongest reported result is Pionono at unweighted kappa 0.758 Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation. The Latent Doctor Model reports a KL divergence of 0.461 to the multi-expert label distribution The Latent Doctor Model. Rater-capped sampling is orthogonal to these models, so we apply it on top of Pionono and on top of the Latent Doctor Model rather than only competing against them.
We also include two anchor-free filters: DGN-S ranking and Dawid-Skene. Each removes the same number of raters as the best conformity quartile. Every method gets the same backbone, the same number of training steps, and the same hyperparameter search budget (an equal number of configurations). All tuning uses held-out raters that never appear at test time.
Two ablations isolate the key ingredient. The first caps rater share $\pi_j$ alone while ignoring disagreement $\delta_j$. The second compares rater bagging with image bagging at an equal number of ensemble members and equal compute. Beating the baselines means matching or exceeding conformity filtering on held-out-rater F1 without using an anchor, and exceeding SVGPCR's 0.7048 kappa on CrowdGleason under the same features and supervision.
Why not just…?
Why not just use majority vote and apply ordinary stability theory? Majority vote hides the rater structure without removing it. If one prolific rater is idiosyncratic, the consensus labels on their many images still carry their bias. The reported CrowdGleason gap between majority vote (0.6576) and SVGPCR (0.7048) suggests the information majority vote discards matters. The Latent Doctor Model paper also reports that majority-vote models stay overconfident on borderline cases. Our scaling experiment includes majority vote at each $m$, which will show whether it inherits the same $1/m$ dependence.
Why not just filter raters by conformity, as Variability Matters does? This is the real threat, and it may win when the evaluation panel resembles the anchor. Our account predicts that filtering improves agreement with a panel like the anchor partly by changing which target the model learns. Under evaluation on randomly held-out raters, its advantage over rater-capped sampling should shrink or reverse. We test this by evaluating both on the original three-pathologist validation panel and on the held-out-rater protocol. If filtering still wins on held-out raters, our capping rule adds nothing, and we will report that.
Why not just add weight decay, dropout, or early stopping? These act equally on every sample. The bound says the damage comes from specific raters with large $\pi_j\delta_j$. SGD Generalizes Better Than GD (And Regularization Doesn't Help) shows that explicit $\ell_2$ regularization cannot fix instability that comes from the algorithm itself, which counts against expecting uniform shrinkage to fix a structured problem. We sweep weight decay and stopping time at the same tuning budget. If the best-tuned regularized pooled model matches rater-capped sampling, the rater-level view is not needed in practice.
Why not just run leave-one-rater-out validation and drop the raters whose removal helps? With a trusted labeled validation set, this brute-force data-valuation approach could do as well as our rule. It needs $m$ retrainings and gold labels, and it cannot be used on the 120-rater benchmark without enormous compute. We run it on Gleason 2019 (six raters) and CrowdGleason (seven residents) as an upper reference. There we check whether cheap $\pi_j\delta_j$ estimates recover its rater ranking.
Proposed Experiments
The decisive check is the fixed-budget scaling study on the 120-pathologist benchmark, evaluated against held-out raters. If error to unseen pathologists does not improve with $m$ at fixed $n$, the central claim is wrong. A second check is whether measured rater-twin divergence correlates with $\pi_j\delta_j$ across raters in deep networks. We test this on DeepLabv3+ for cell detection and on a ResNet18 for CrowdGleason. If the correlation is weak, the bound is not guiding anything real.
The third check is head-to-head. Rater-capped sampling must match or beat anchor-based conformity filtering on held-out-rater F1. On CrowdGleason it must exceed 0.7048 quadratic kappa with matched supervision. On Gleason 2019, adding it on top of Pionono must not drop below 0.758 kappa, and it should improve leave-one-pathologist-out kappa. We use the ISUP biopsy set, graded by 20 pathologists, to test whether capping preserves the Latent Doctor Model's calibrated disagreement. The reference point there is an entropy Spearman correlation of 0.728, measured against held-out pathologists rather than the full panel.
References
- Train faster, generalize better: Stability of stochastic gradient descent
- Variability Matters: Evaluating Inter-Rater Variability in Histopathology for Robust Cell Detection
- Leave One Out Error, Stability, and Generalization of Voting Combinations of Classifiers
- Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernel
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The CrowdGleason dataset: Learning the Gleason grade from crowds and experts
- DGN: Disagreement Graph Networks for Learning from Multiple Annotators
- SGD Generalizes Better Than GD (And Regularization Doesn't Help)