Pathologists who grade the same prostate biopsy often disagree. Much of this disagreement is not random error. Tumors vary along a continuum, and each pathologist cuts that continuum into grades at slightly different places. We propose to model this directly. Each tissue image receives one continuous "malignancy" value, and each pathologist gets their own set of grade cutoffs on that shared scale. A case is contested when its malignancy value lands where pathologists' cutoffs differ. The malignancy function gets a prior taken from the infinite-width limit of a neural network: as a network's layers grow infinitely wide, its outputs behave like a Gaussian process (GP), a distribution over functions fully described by a similarity kernel between inputs. Using this limit, the model can be fit with standard GP machinery for ordered outcomes rather than through variational autoencoders or ensembles of trained networks.
This matters for two reasons. Current tools either collapse disagreement into a majority vote, which makes models overconfident on borderline cases, or represent each rater's behavior with a learned latent code or a fixed confusion matrix that is hard to interpret. A threshold model states something clinicians can check: pathologist B calls Gleason 4 earlier than pathologist A. It also separates two kinds of uncertainty that current methods blur. One is the model not knowing where a case sits on the scale (GP posterior variance). The other is pathologists genuinely disagreeing about where to cut (spread of cutoffs).
Core Claim
A single latent malignancy function, combined with rater-specific ordinal cutoffs and a small rater-specific deviation function, should predict multi-pathologist label distributions at least as well as the conditional latent-variable models now used for this purpose. It should also need far fewer raters per training case. The neural-network-limit kernel should be at least as good as a generic kernel on the same features. Concretely, rater $r$'s probability of assigning grade $c$ to image $x$ is
$$P(y_r = c \mid x) = \Phi\!\left(\frac{b_{r,c} - h_r(x)}{\sigma_r}\right) - \Phi\!\left(\frac{b_{r,c-1} - h_r(x)}{\sigma_r}\right), \qquad h_r(x) = f(x) + g_r(x).$$
Here $\Phi$ is the standard normal cumulative distribution and $b_{r,c}$ is rater $r$'s ordered cutoff between grades $c$ and $c+1$. The shared malignancy function $f$ has an infinite-width kernel prior $K$. The term $g_r \sim \mathrm{GP}(0, \lambda K)$ is a small rater-specific deviation that lets a rater be systematically harsher on some morphologies than on others. The scalar $\sigma_r$ is the rater's own noise level.
If cutoffs across the rater population are written as $b_{r,c} = \beta_c + \delta_{r,c}$ with $\delta_{r,c} \sim \mathcal{N}(0, \tau_c^2)$, and $g_r$ is ignored, then an infinitely large panel calls grade above $c$ with probability $\Phi\big((f(x) - \beta_c)/\sqrt{\sigma^2 + \tau_c^2}\big)$. This follows from integrating a probit over a Gaussian shift. The parameter $\tau_c$ is therefore a direct, per-grade-boundary measure of how much the profession disagrees. Because the expected panel distribution has a closed form, we can also quantify how noisy a finite panel's empirical distribution is as a reference.
Relation to Prior Work
Two existing works motivate the proposal and define the gap. The Latent Doctor Model treats disagreement as signal. It uses a conditional variational autoencoder with a set encoder over each doctor's past labels, and reports a KL divergence of 0.461 on Gleason2019 and a Spearman correlation of up to 0.728 between its predictive entropy and the entropy of 20 pathologists on ISUP grading. Its rater representation is an uninterpretable latent vector trained with an approximate objective, and it ignores the ordering of grades. The CrowdGleason dataset applies sparse variational GPs on frozen ResNet18 features to crowd labels from seven residents and reaches a quadratic kappa of 0.7048 against 0.6576 for majority voting. Its authors state that their per-annotator confusion matrices are feature-independent, and the matrices treat the four grades as unordered categories. We replace both rater models with cutoffs on an ordered scale plus a feature-dependent deviation. We keep CrowdGleason's frozen-feature design so the comparison is fair.
The method borrows from three sources. Gaussian Processes for Ordinal Regression supplies the probit threshold likelihood and its Laplace and expectation-propagation inference. That work assumes one consensus label per case and a generic radial basis function (RBF) kernel; we give each rater their own thresholds and swap in network-limit kernels. Gaussian Process Behaviour in Wide Deep Neural Networks and Deep Convolutional Networks as shallow Gaussian Processes supply the recursive kernels for wide fully connected and convolutional networks. Scale Mixtures of Neural Network Gaussian Processes shows that placing an inverse-gamma prior on the readout scale turns the limit into a heavy-tailed Student-t process. We use the same device on the per-rater noise $\sigma_r$ so that erratic raters are down-weighted rather than distorting $f$.
Uncertainty in Neural Networks: Approximately Bayesian Ensembling plays a different role. It shows that anchored ensembles of finite networks converge to the GP limit as width and ensemble size grow. This gives a principled finite-network counterpart to compare against.
Two findings constrain the design. The infinite-width limit learns no features, and the file's own numbers show limit kernels trailing trained networks on images: a residual attention kernel reaches 80.72% on CIFAR-10 in Infinite attention. We therefore do not put the kernel on raw pixels in the main arm. Instead it acts as the infinite-width limit of a classification head on pretrained tissue features. Variability Matters finds a Fleiss' kappa of only 0.43 across 120 pathologists and shows that low-conformity annotators hurt training. This motivates the heavy-tailed per-rater noise.
Research Program
The model is built in three layers.
The first layer is the kernel. We compute the infinite-width kernel of a ReLU network head of depth one to three on frozen ResNet18 features, matching CrowdGleason. A second arm uses frozen features from a pathology-pretrained backbone. A third arm applies the convolutional limit kernel directly to downsampled patches as a test of whether architecture alone carries useful structure; we expect this arm to be weak.
The second layer is the likelihood above, with cutoffs constrained to be ordered through a cumulative-softplus reparameterization. The deviation functions $g_r$ share the kernel $K$ and are scaled by a learned $\lambda$, which makes this a multi-output GP with one output per rater plus the shared one.
The third layer is inference. Exact Laplace or expectation propagation is used on Gleason2019 (244 cores) and ISUP (100 slides, represented by mean-pooled patch features; this is a simplification we state openly). Inducing-point variational inference is used on CrowdGleason's 19,077 patches, mirroring that paper's scale handling. Kernel hyperparameters, cutoffs, $\lambda$ and the noise priors are fit by maximizing the approximate marginal likelihood.
The analyses ask three things. First, does the fitted $\tau_c$ concentrate at clinically known contested boundaries, such as Gleason 3 versus 4? Second, do individual rater cutoffs predict that rater's held-out labels? Third, does the GP posterior variance of $f$ rise on the external SICAPv2 cohort while the cutoff-driven disagreement stays put? That last result is what separating the two kinds of uncertainty should produce.
Baselines
The naive reference is a finite ResNet18 head on the same frozen features. We run it twice: once trained on majority-vote labels, and once with cross-entropy against each case's empirical soft label distribution. The soft-label variant is the most dangerous simple competitor for KL divergence, because it targets the evaluation quantity directly. Both use the same splits and the same hyperparameter search budget as our model, measured in configurations tried. CrowdGleason reports a majority-vote quadratic kappa of 0.6576.
The strongest existing method for predicting label distributions is the conditional latent doctor model from The Latent Doctor Model. It reports a KL divergence of 0.461 on Gleason2019 and an entropy Spearman correlation of up to 0.728 on ISUP. We report against those published figures and also reimplement the model on our splits with the same frozen features, because protocol differences could otherwise explain any gap. Beating it means lower KL and higher entropy correlation on our splits, with confidence intervals from repeated splits that exclude zero difference. On the published-protocol comparison we aim to beat 0.461 and 0.728. Its deep-ensemble comparison, one network per doctor, is also run.
For crowd-label aggregation, the strongest baseline is SVGPCR from The CrowdGleason dataset at a quadratic kappa of 0.7048 on CrowdGleason. SVGPMIX uses extra expert labels and reaches 0.7276 on CrowdGleason and 0.7814 on SICAPv2. We compare our crowd-only model to SVGPCR. We compare to SVGPMIX only after adding the same expert labels as an additional, low-noise "rater". Beating SVGPCR means a quadratic kappa above 0.7048 using identical ResNet18 features, so any gain is due to the rater model and not the representation.
Four ablations isolate the key ingredients. Replacing the infinite-width kernel with an RBF kernel on the same features tests whether the network limit matters at all. Collapsing all raters to one set of cutoffs, which recovers Chu and Ghahramani's consensus model, tests whether personal cutoffs matter. Dropping $g_r$ tests whether feature-dependent rater behavior adds anything beyond threshold shifts. Replacing the ordinal likelihood with CrowdGleason-style confusion matrices tests whether the ordering of grades matters. Finally, anchored finite ensembles of the same head at increasing widths and ensemble sizes show whether a trained finite network approaches, exceeds or falls short of its limit on this data.
Why not just…?
Why not just train a network on the soft label distributions? This is the most serious threat to our KL numbers, and we cannot rule it out from the literature. The latent doctor paper shows consensus training is overconfident, but the file reports no soft-label baseline number. Soft-label training cannot predict a specific pathologist's call, cannot be trained when each case has only one or two raters, and gives no rater-level parameters. The panel-subsampling experiment below is designed to show where it breaks. If it matches us on full-panel KL, we will say so and rest the case on the low-panel and per-rater results.
Why not just use an RBF kernel and skip infinite-width theory? This is plausible. On frozen features the network-limit kernel may behave much like a well-tuned RBF kernel. The RBF ablation settles the question. If the two tie, the contribution reduces to the threshold rater model, and the infinite-width part remains useful only as the exact limit that ties the GP to the finite-head and ensemble baselines.
Why not just fine-tune the backbone end to end? Feature learning is exactly what the limit kernel lacks. Fine-tuned networks beat fixed-feature classifiers on accuracy, as GasHisSDB shows with 96.47% for VGG16 against 86.08% for colour-histogram forests (GasHisSDB). But Gleason2019 has 244 cores and ISUP has 100 slides. With so few labeled cases and noisy, multi-rater targets, end-to-end fine-tuning risks overfitting the consensus. We include a fine-tuned soft-label baseline on CrowdGleason, where the data volume makes it fair, and report whether it wins.
Why not just keep confusion matrices or the latent doctor code? The rater-model ablation compares them directly. Our prediction is that ordered cutoffs win most clearly at adjacent-grade boundaries and when each rater contributes few labels, because a cutoff vector has far fewer free parameters than a four-by-four confusion matrix.
Decisive Experiments
The central test uses ISUP's 20-rater panel. We train using only $k \in \{1, 2, 3, 5\}$ randomly chosen raters per slide and evaluate against the held-out remaining raters. We measure KL divergence to their empirical distribution and the Spearman correlation between predictive entropy and their entropy. The closed-form infinite-panel expression lets us report how much of the remaining error is sampling noise in a finite reference panel. The proposal succeeds if, at $k \le 3$, the personal-cutoff model beats both the soft-label network and the reimplemented latent doctor model, and if it matches or beats 0.728 at full panel.
On Gleason2019, the target is KL divergence below 0.461. On CrowdGleason, the targets are a quadratic kappa above 0.7048 with crowd labels only, plus an out-of-distribution evaluation on SICAPv2.
A per-rater test holds out 20% of each rater's labels and scores log-likelihood of that specific rater's calls. This is the question the latent doctor model's set encoder was built to answer.
Finally, the fitted $\tau_c$ values and rater cutoffs are checked qualitatively against known contested boundaries. If disagreement does not concentrate at the Gleason 3/4 boundary, that is evidence against the shared-scale assumption itself, and we will report it as such.
References
- The Latent Doctor Model for Modeling Inter-Observer Variability
- The CrowdGleason dataset: Learning the Gleason grade from crowds and experts
- Gaussian Processes for Ordinal Regression
- Gaussian Process Behaviour in Wide Deep Neural Networks
- Deep Convolutional Networks as shallow Gaussian Processes
- Scale Mixtures of Neural Network Gaussian Processes
- Uncertainty in Neural Networks: Approximately Bayesian Ensembling
- Infinite attention: NNGP and NTK for deep attention networks
- Variability Matters: Evaluating Inter-Rater Variability in Histopathology for Robust Cell Detection
- GasHisSDB: A new gastric histopathology image dataset for computer aided diagnosis of gastric cancer