Pathologists who score the same slide often report different numbers. One calls a tumor 15% Ki67-positive and another calls it 30%; one counts four mitoses per ten high-power fields and another counts seven. Ki67 is a protein that marks dividing cells, and the percentage of stained nuclei estimates how fast a tumor grows. Current machine learning treats these disagreements in one of two ways. Either it averages them away into a consensus label, or it learns a free-floating "rater style" vector that reproduces the spread without explaining it. We propose a third option. Every rater looks at the same underlying tissue, so we model each rater as a small, physically meaningful measurement operator applied to one shared, scale-aware map of the tissue. The operator records how faint a stain must be before that rater calls a nucleus positive, and whether the rater counts in the densest "hotspot" or averages over the whole field. A density network (a model that outputs a per-pixel map whose sum over any region gives the object count) learns the shared map. Each rater's reported number is supervision for that map, passed through the rater's own operator.
The idea matters for three reasons. First, the operator parameters are interpretable, so a learned rater can be checked against morphology. Prior rater-modeling work lists this as unresolved. Second, the model predicts which cases will produce disagreement: those with many borderline-intensity nuclei, or with uneven proliferation across the field. It does not simply predict that disagreement exists. Third, it lets archived clinical reads, the single numbers pathologists write into reports, serve as weak multi-observer supervision for dense counting. Those reads are currently thrown away because they are noisy and observer-specific.
Core Claim
Inter-observer disagreement in counting and proportion-based grading is largely structured. It comes from rater-specific thresholds and rater-specific spatial aggregation scales applied to a common continuous tissue quantity. A model that factors the problem this way should beat three alternatives:
- Consensus regression, on point estimates.
- A per-rater constant offset, on predicting case-level disagreement.
- An unstructured latent-rater model, on fitting new raters from few readings and on interpretability.
The four directions play unequal roles. Scale variation supplies the core mechanism: a density map with local-to-global consistency. Inter-observer variability is the phenomenon being explained. Multi-task learning is the training structure, with one shared backbone and many tiny per-rater heads whose disagreements would otherwise conflict on shared parameters. Multimodal document representation is the data channel through which, in the second phase, archived report values enter training with extraction uncertainty attached.
The Model
A shared encoder-decoder produces two maps from an image $x$: a nucleus density $D(u)$ at location $u$, and a stain-intensity score $s(u)$ for the nuclei there. For a Ki67 reading, rater $r$ has a positivity threshold $\tau_r$ and a spatial concentration $\beta_r$. The concentration parameter sets a softmax weighting $w_{\beta_r}(u)$ over candidate fields. When $\beta_r$ is large, the weighting picks out the most proliferative field, which is hotspot scoring. When $\beta_r$ is near zero, it averages over the whole slide. The rater's predicted index is
$$\hat y_r(x) = \frac{\int w_{\beta_r}(u)\, D(u)\, \sigma\!\big((s(u)-\tau_r)/T\big)\, du}{\int w_{\beta_r}(u)\, D(u)\, du},$$
where $\sigma$ is the logistic function and $T$ is a fixed temperature that keeps the threshold differentiable.
For Gleason grading, the shared map is instead a per-pixel distribution over growth patterns. The Gleason score sums the two most prevalent patterns, and the ISUP grade groups those sums into five classes. The rater's operator holds per-pattern calling thresholds plus the area fractions at which a minor pattern starts to count. Grade then follows deterministically from the thresholded area fractions, because the clinical grading rules are themselves defined over pattern proportions.
Training combines two losses. The first is a local density loss on whatever point annotations exist. The second is a per-rater reading loss on observed numbers:
$$\mathcal{L} = \sum_{g \in \mathcal{G}} \Big|\textstyle\int_g D - c_g\Big| + \lambda \sum_{(x,r,y)} \ell\big(\hat y_r(x), y\big).$$
In the first term, $\mathcal{G}$ is a set of grid cells at several scales and $c_g$ is the annotated count in cell $g$. This grid term follows Towards Locally Consistent Object Counting with Constrained Multi-stage CNNs, which showed that controlling subregion errors bounds global count error and stops local undercounting from being hidden by spurious density elsewhere. In the second term, $\ell$ is a likelihood suited to the reading type: Gaussian for indices, interval-censored when only a range is reported, categorical for grades. The weight $\lambda$ balances the two terms.
The point annotations matter for identifiability. Without them, a higher threshold could be traded against a denser map. Anchoring $D$ locally is what makes $\tau_r$ and $\beta_r$ meaningful.
Raters with many readings get their own $(\tau_r, \beta_r)$. For a new rater, an encoder that reads a set of examples and is insensitive to their order (a DeepSet, borrowed from The Latent Doctor Model) infers the operator from a handful of that rater's readings.
Relation to Prior Work
The closest prior work models disagreement, and it motivates the gap. The Latent Doctor Model uses a conditional variational autoencoder: an unstructured latent variable stands for the rater, and the model samples plausible grades on patches. It matches expert-panel distributions on Gleason2019 and ISUP. Its rater latent has no physical meaning, and it does not touch spatial counting. Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle learns "expertness maps" that weight rater segmentations so that downstream diagnosis improves. Its authors note that the learned ground truth then depends on the diagnostic network, and that alignment with biomarkers is open. We keep raters separate and explain them, rather than reweighting them toward a downstream target. Supporting Mitosis Detection AI Training with Inter-Observer Eye-Gaze Consistencies aggregates gaze from 14 non-experts into consensus centroids. We use the same per-observer signals but model them instead of averaging them.
The density side supplies the method. Preserving dense features for Ki67 nuclei detection (UV-Net) keeps high-resolution features through its encoder for overlapping nuclei, and we adopt it as the backbone. CP-CNN and TEDnet show that pixel-wise squared error alone blurs density maps and that multi-scale constraints fix this. That finding motivates the multi-scale grid term.
On the multi-task side, RotoGrad reduces conflict between tasks by learning a per-task rotation of shared features. Our per-rater operators play the same conflict-absorbing role, but with interpretable low-dimensional parameters instead of rotations. RotoGrad therefore becomes a direct baseline for the claim that structure helps.
On the document side, Multimodal Structured Foundation Models for Noisy Documents shows that layout-aware key-value extraction from noisy clinical reports, with OCR reliability fed in as an input, beats a much larger text-only model. Synoptic reporting by summarizing cancer pathology reports using large language models extracts synoptic elements from narrative reports. We reuse such extractors rather than building one, and pass their confidence into $\ell$ as a per-reading noise scale.
Research Program
Phase one uses public multi-observer data with no documents.
- Gleason2019: 244 tissue cores, six pathologists, pixel-level pattern maps. Here the pattern-proportion operator can be fit and checked against each pathologist's own maps.
- ISUP Grading set: 100 biopsies, 20 pathologists. This is the test of whether operators inferred from a few readings generalize to held-out cases.
- Meningioma eye-gaze set: 800 high-power fields. Per-observer fixation blobs give per-observer counts, so a count-threshold operator can be fit for each of the 14 observers.
- Ki67: the SMH/Deepslides centroid patches anchor the local density. The Protein Atlas cores, whose proliferation index is reported only as ranges, test interval-censored readings. The UHN cores, with a per-patient index averaged over cores, test aggregation.
Phase two requires an institutional cohort, and we flag it as the main data risk. The cohort would pair archived slides with the pathology reports written on them, the report author's identity, and the scanned report pages. A layout-aware extractor produces readings with confidences, and these train the same model. The question is whether thousands of free observer-specific reads, each just one number, improve density estimation and rater modeling over a small expert-annotated set alone.
Baselines
The naive reference is consensus regression. UV-Net is trained on point annotations, its index is read off directly, and every rater's reading is scored against that single prediction, which implies zero predicted disagreement. On Protein Atlas cores, UV-Net reports proliferation-index estimation accuracy of 0.7631 (UV-Net). On SMH/Deepslides it reports F1 of 0.8333 for Ki67-negative and 0.8201 for Ki67-positive nuclei. We must at least match these, since the shared map should not degrade.
The strongest existing method for disagreement is the Conditional Latent Doctor Model. It reports a KL divergence of 0.461 to the expert panel on Gleason2019, and a Spearman correlation of up to 0.728 between model entropy and the entropy of the 20 pathologists on ISUP (The Latent Doctor Model). We rerun it on our splits with the same backbone features, the same training images, and an equal hyperparameter-search budget. Beating it means a lower KL on Gleason2019 and a higher entropy correlation on ISUP. We also require a clear advantage when each held-out rater has only five to ten readings, which is where the structured operator should matter most. For the Ki67 and gaze-count settings, no reported result was found on Lacuna for a multi-rater method, so LDM adapted to continuous readings is our own reimplementation.
The ablations isolate the key ingredient.
- RotoGrad multi-head regression: one head per rater on the shared backbone, using RotoGrad for balancing. This tests whether any conflict-absorbing per-rater head suffices.
- Offset model: each rater is the consensus prediction plus a learned constant.
- Threshold only: the operator with $\tau_r$ but no $\beta_r$.
- Scale only: the operator with $\beta_r$ but no $\tau_r$.
- Report-reading loss removed: this applies in phase two.
All variants share data, backbone, compute, and tuning budget. Beating them means better held-out per-rater likelihood. It also means better correlation between predicted and observed per-case disagreement, measured as the spread of readings across raters on each case.
Why not just…?
Why not just regress the consensus and report a confidence interval? This misses the structure of disagreement. Under consensus training, uncertainty reflects model error, not rater behavior. The LDM paper shows majority-vote training becomes overconfident on borderline cases. Our naive baseline measures this directly: it predicts no case-specific spread, so its correlation with observed per-case spread is the reference the full model must exceed.
Why not just give each rater a constant offset? This is the most serious threat. If raters differ mainly by a fixed bias, our operators add nothing. The decisive difference is that the threshold operator predicts disagreement that grows with the fraction of borderline-intensity nuclei, and the scale operator predicts disagreement that grows with how uneven proliferation is across fields. A constant offset predicts uniform disagreement. We stratify held-out cases by these two image statistics, both computed from the shared map. If disagreement does not vary with them, we will report that the offset model suffices and the central claim fails.
Why not just use an unstructured latent rater, as in the Latent Doctor Model? It may fit the panel distribution equally well when raters have abundant data. Our bet is on two things. One is sample efficiency for new raters, because two physically constrained parameters need fewer readings than a free latent. The other is interpretability: we can test whether a fitted $\tau_r$ ranks raters the same way their own nucleus-level calls do on SMH-style patches. The few-shot ISUP comparison and the Gleason2019 check against each pathologist's own maps settle both points.
Why not just extract report values with a text-only language model? For a single Ki67 percentage in narrative text, that may be enough, and we will use it if extraction error is negligible. Layout-aware extraction matters only for tabular synoptic forms. There, Chargrid shows text-only sequence models near 0% on multi-line table items, and the noisy-clinical-documents encoder beats a 0.6B text-only model by 31.1 F1 on key-value pairing. We will measure extraction F1 for both extractors on a hand-checked sample. Then we feed each into training, so the choice is judged by downstream rater-model quality and not by assumption.
Decisive Checks
Three results would establish the claim. On ISUP held-out cases, predicted per-case disagreement must correlate with the observed spread among the 20 pathologists better than the offset and RotoGrad baselines. It must also at least match LDM's reported 0.728 entropy correlation. On Gleason2019, the fitted pattern thresholds for each of the six pathologists must reproduce that pathologist's own area-fraction maps better than chance and better than an operator fit on pooled data. On the eye-gaze set, per-observer count operators fit on half the fields must predict each observer's counts on the other half. Per-case error should shrink where the shared map shows few borderline nuclei.
One result would refute the claim: across all three regimes, disagreement shows no dependence on borderline-intensity mass or spatial unevenness. That would mean rater variability here is unstructured noise, and no operator can explain it.
References
- Towards Locally Consistent Object Counting with Constrained Multi-stage Convolutional Neural Networks
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Supporting Mitosis Detection AI Training with Inter-Observer Eye-Gaze Consistencies
- Preserving dense features for Ki67 nuclei detection
- Generating High-Quality Crowd Density Maps Using Contextual Pyramid CNNs
- Crowd Counting and Density Estimation by Trellis Encoder-Decoder Networks.
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Multimodal Structured Foundation Models for Noisy Documents: A 110M Encoder Where Structure Matches Scale on Clinical Key--Value Generation
- Synoptic reporting by summarizing cancer pathology reports using large language models