Computational pathology models are scored against the consensus of a panel of pathologists, and that consensus is itself noisy. A model can fall short because it is weak, or because the reference it is scored against cannot be matched any better. This proposal computes, for each multi-rater pathology benchmark, a ceiling that no model of any size can exceed. The ceiling is a converse bound in the information-theoretic sense: a provable limit on agreement with a K-pathologist consensus, derived from how often pathologists split their votes on each case. The same analysis predicts how much a model gains when it is trained on the consensus of k pathologists instead of one. If the bound is tight, it separates "the model is weak" from "the reference standard is saturated." That distinction currently decides whether a lab spends money on more annotators, bigger models, or biochemical ground truth.
There is a concrete puzzle to start from. Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment trained mitosis detectors on labels from 12 veterinary pathologists. Their F1 scores plateaued at 0.74–0.76 regardless of which annotators supplied the training labels. Training-run variance fell sharply from one annotator (interquartile range 0.038) to three (0.016), and further annotators added only about 0.013 F1. That paper is purely empirical and leaves open whether the plateau reflects model capacity, an unreachable reference, or information discarded by majority voting. The claim here is that both the plateau and the three-rater saturation follow largely from the per-case structure of rater disagreement. Both can be computed from the vote matrix alone, before any network is trained.
Core Claim
Let $X$ be an image region or detection candidate. Let $\pi(X)$ be the probability vector describing how a randomly drawn pathologist would label it, which we call the case's vote profile. Let $Y_K$ be the consensus label formed by majority vote of $K$ pathologists. Assume pathologists' votes depend on the image only through $\pi(X)$ and the identity of each rater. Under that assumption, any predictor $f(X)$ with error rate $e = P(f(X) \neq Y_K)$ over $C$ classes satisfies the Fano-type inequality
$$h(e) + e \log(C-1) \;\ge\; H\big(Y_K \mid \pi(X)\big),$$
where $h$ is the binary entropy function and $H(Y_K \mid \pi)$ is the uncertainty left in the consensus label once the vote profile is known. The inequality holds because seeing the image cannot reveal more about future votes than the vote profile does. It therefore caps every model, however large.
For detection and F1 the inequality is not used directly. The optimal predictor is a threshold on $P(Y_K = 1 \mid \pi)$, and its expected F1 can be computed exactly given $\pi$. We call this the reference ceiling.
The training-panel effect follows the same logic. A model trained to convergence on $k$-rater consensus labels learns at best $P(Y_k \mid X)$. Its agreement with a separate $K$-rater test consensus is therefore bounded by a transfer ceiling $A(k, K)$, which depends only on the distribution of vote profiles across cases. The informative quantity is $I(Y_k ; Y_K \mid \pi)$, the information that a $k$-rater training label carries about the test consensus. That information saturates quickly in $k$ when most cases are clear and a minority are near-ties. The prediction is that the "three raters is enough" finding is a property of the data's ambiguity profile, not of RetinaNet.
Relation to Prior Work
Two lines of work supply the method. Rethinking Fano's Inequality in Ensemble Learning tightened Fano lower bounds for model ensembles by adding a combination-loss term, reaching a 0.984 Pearson correlation with actual error on GLUE/SuperGLUE. It assumes clean gold labels and treats the voters as models. We reuse the Fano machinery and the idea that voting discards information. Here, though, the voters form the reference rather than the predictor, and the gold label is itself a vote.
Label Correction of Crowdsourced Noisy Annotations with an Instance-Dependent Noise Transition Model derives a Bayes-error bound for crowdsourced aggregation under case-dependent noise, but on CIFAR and LabelMe. Its goal is to correct training labels, not to bound agreement with a finite-panel reference. We borrow its rater-specific, case-dependent noise view to relax the assumption that raters are interchangeable.
Three papers motivate the proposal by showing the gap empirically: Wilm et al. above, The Latent Doctor Model for Modeling Inter-Observer Variability, and Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation. The Latent Doctor Model is a conditional variational autoencoder with per-doctor style codes, reporting KL divergence 0.461 on Gleason 2019. Pionono represents each annotator as a Gaussian latent distribution, reporting Cohen's κ 0.758 on Gleason 2019 and 0.711 on a 25-rater breast set. Both model the disagreement well, but neither says how close their numbers are to what is attainable.
One work constrains the design. L_DMI shows information-theoretic robustness to label noise only when noise is independent of the case. Pathologist noise concentrates on borderline morphology (see the gastric dysplasia study), so we condition on the per-case vote profile rather than on a single class-level confusion matrix.
The file suggests one alternative route, modeling panels as cut-set problems as in Generalized Cut-Set Bounds for Broadcast Networks. We deliberately do not take it. Those bounds address rates over physical networks. A panel of conditionally independent raters is a much simpler star structure, where Fano's inequality and data processing already give the tight statement.
Research Program
The work has three parts.
The first part is an estimator for $H(Y_K \mid \pi)$, the reference ceiling, and the transfer curve $A(k, K)$ from a finite vote matrix. Plugging raw vote fractions in for $\pi$ is biased. With $n$ raters, estimated profiles are lumpy, and taking a maximum over a noisy estimate inflates apparent certainty. We therefore fit a hierarchical model of vote profiles: a Beta or Dirichlet mixture across cases, with Dawid–Skene-style per-rater biases (each rater has their own confusion pattern). We then integrate the ceiling over the posterior on $\pi$. This yields both a point estimate and a credible interval. Split-panel resampling serves as an internal consistency check: estimate the profile from half the raters, score against a consensus from the other half.
The second part is the analytic shape of $A(k, K)$. We expect a fast rise followed by saturation, with the knee set by the fraction of near-tie cases. The analysis would also predict Wilm et al.'s otherwise surprising result that the three least agreeable raters (F1 0.756) train as well as the three most agreeable (0.751). Under the model, what matters is whether a rater subset's majority matches the full-panel majority on ambiguous cases. Pairwise agreeableness is a poor proxy for that.
The third part tests whether models actually sit at the ceiling. We retrain detectors and graders with increasing capacity and see whether any exceed the bound, which would falsify the conditional-independence assumption, or approach it.
Baselines
The naive reference point is human-level agreement: each pathologist scored against the consensus of the remaining raters, so the rater is not voting on their own reference. Wilm et al. report 0.64–0.85 F1 for individual pathologists, but against a 12-rater consensus that includes them, which inflates the figures. We recompute these leave-one-out, on the same splits, as the do-nothing ceiling that practitioners informally use.
The strongest competing ceiling estimator is the plug-in split-panel estimator. It sets each case's profile to its raw vote fraction, computes the optimal-threshold F1 or accuracy against a disjoint rater subset, and needs no modeling. It will be run on identical vote matrices with the same rater splits and the same number of resamples. Our estimator beats it if it is closer to ground truth on synthetic panels with known $\pi$ at small panel sizes (n = 6, as in Gleason 2019) and at least as close at n = 12–20. A second analytic competitor is the crowdsourced Bayes-error bound of the GLMM label-correction paper, which we instantiate on the same data.
The strongest existing models define the empirical side. For mitosis, the RetinaNet results of Wilm et al. give F1 0.74–0.76 against the 12-rater majority. For Gleason 2019 there are Pionono (κ 0.758) and CLDM (KL 0.461). For ISUP there is the Latent Doctor Model's entropy correlation (Spearman up to 0.728 with 20 pathologists). For 3k Buds, the Latent Doctor Model is reported only as "competitive" with deep ensembles; no reported numeric result was found on Lacuna for 3k Buds. Where code is unavailable, we retrain comparable architectures under a fixed compute and tuning budget, matched across panel sizes.
Two ablations isolate the key ingredients. The first removes per-rater bias and treats raters as interchangeable, which tests whether rater-specific modeling matters for the bound. The second removes the hierarchical smoothing and reverts to plug-in profiles, which tests the debiasing.
Why not just…?
Why not just use the leave-one-out pathologist score as the ceiling? Single raters bundle two things: the unavoidable disagreement in the reference and that rater's own idiosyncrasies. The best leave-one-out pathologist is therefore a lower estimate of what a profile-aware predictor can reach. The worst is a meaningless one. The reported spread of 0.64–0.85 shows there is no single human number to use. Our experiments report both quantities side by side. If the leave-one-out maximum consistently matches our ceiling, the simpler heuristic is vindicated, and we would say so.
Why not just compute the plug-in split-panel ceiling and skip the theory? This is the real threat. With 12 or 20 raters, the plug-in estimate may be nearly as good, and the synthetic-panel experiment is designed to show exactly where it stops being adequate. We expect it to break down at 6 raters, where vote fractions take only seven values. The plug-in estimate also cannot extrapolate. It cannot predict the transfer curve for panel sizes not collected, or say how many raters a new study needs. If it matches our estimator everywhere, the contribution shrinks to the transfer-curve analysis and the capacity check, and the note will report that.
Why not just train bigger models and see whether the plateau moves? A capacity sweep alone cannot distinguish a saturated reference from a hard optimization problem, because both look like a plateau. We run the sweep, but as a test of the bound rather than a substitute for it. Models plateauing at the computed ceiling means the reference is the bottleneck. Models plateauing well below it means representation is the bottleneck, and annotation budgets should go elsewhere.
Why not just estimate latent truth with Dawid–Skene or the Latent Doctor Model? These methods estimate labels or label distributions, not attainable agreement. We use the Dawid–Skene structure inside our estimator. The Latent Doctor Model is a model to be scored against the ceiling, not a ceiling itself.
Proposed Experiments
The first check is synthetic. We simulate panels with known vote profiles, ambiguity mixtures, and rater biases at n = 6, 9, 12, and 20. We then verify that the estimated ceiling and transfer curve recover the true values, and that the plug-in estimator does not at small n.
The second check is the mitosis dataset: 12 raters, 50 canine mast cell tumor images, and the 35/5/10 split of Wilm et al. Candidates are the union of all raters' annotations. The decisive question is whether the computed F1 ceiling against the at-least-7-of-12 majority lands at or just above 0.74–0.76. We also test whether $A(k, 12)$ reproduces the jump from one to three raters, the roughly 0.013 gain beyond three, and the near-equality of the least and most agreeable triplets. We retrain RetinaNet with ResNet18 and larger backbones to see whether any model exceeds the bound. Exceeding it would expose a violated assumption, most plausibly raters sharing visual cues beyond $\pi$.
The third check covers Gleason 2019 (6 raters) and ISUP (20 raters, 6 ordinal grades). We compute accuracy and κ ceilings against majority labels, plus the irreducible KL floor between a perfectly calibrated predictor and the finite-panel empirical distribution. That floor is nonzero simply because 6 votes estimate the distribution coarsely. Comparing it to CLDM's 0.461 and Pionono's 0.758 shows how much headroom those methods leave.
On 3k Buds and the 25-rater breast set we repeat the transfer-curve prediction by subsampling raters, as an out-of-sample test of the saturation analysis.
The proposal fails if published or retrained models routinely exceed the ceiling, or if the predicted transfer curve misses the observed one on held-out rater subsets. It succeeds if the ceiling explains the mitosis plateau and correctly predicts which benchmarks still have room to improve.
References
- Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment
- Rethinking Fano's Inequality in Ensemble Learning
- Label Correction of Crowdsourced Noisy Annotations with an Instance-Dependent Noise Transition Model
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- L_DMI: A Novel Information-theoretic Loss Function for Training Deep Nets Robust to Label Noise
- Generalized cut-set bounds for broadcast networks