Do radiologists disagree where the tumor really is? Checking MRI contour disagreement against whole-mount histology

When several radiologists outline the same prostate tumor on MRI, they rarely agree. Published overlap between readers sits around a Dice score of 0.48–0.52 (Dice measures shared area; 1 is perfect agreement). The machine-learning response has been to model the disagreement itself: to learn a distribution of plausible contours instead of one "true" mask. Those models reproduce human disagreement, but nobody has checked whether that disagreement means anything biologically. There are two possibilities. In the first, the ring of voxels where only some readers mark tumor is where cancer actually extends beyond what everyone agrees on. In the second, the disagreement is perceptual noise and the tumor's true extent is simply larger than all of them draw. Prostatectomy supplies the test. The removed gland can be cut into whole-mount slides, examined under the microscope, and warped back onto the pre-surgical MRI.

This note proposes a histology-anchored study of multi-radiologist disagreement, plus a small calibration model built on it. The study's central design choice is to account for two error sources that would otherwise contaminate the answer. The first is registration error, the few-millimetre misplacement left over after digitally warping shrunken, sliced tissue back onto MRI. The second is disagreement among pathologists outlining tumor on the slides. The deliverable is a map from "how many radiologists marked this voxel, and where it sits relative to their contours" to "probability that microscopic cancer is present," scored with these errors in the loop. The main question it answers is whether reader disagreement carries information beyond a uniform safety margin around the consensus contour.

Core Claim

The prior work motivates a specific claim. Explainable AI for CNN-based prostate tumor segmentation in multi-parametric MRI correlated to whole mount histopathology trained a 3D U-Net on single-radiologist contours and tested it on 15 prostatectomy patients with registered histology. It found that radiologist contours systematically underestimate histological tumor volume, so models trained on them inherit a conservative bias. It did not model multiple readers.

The multi-rater models go further on disagreement but stay inside human annotations. Variational inference for quantifying inter-observer variability samples alternative contours from a small learned latent space. Pionono gives each rater a Gaussian in latent space, separating systematic bias from inconsistency. The Latent Doctor Model conditions on a support set of a doctor's past annotations. Marin et al. turns repeated contours from four readers into ordered confidence levels. None of them can say which reader is right, because none has an independent physical reference.

We claim that, after correcting for registration blur and pathologist variability, the probability of histological tumor rises monotonically across radiologist agreement tiers. We further claim that disagreement predicts where tumor spills past the consensus contour better than distance from the contour alone does. If the second claim fails, the correct clinical output is a fixed margin rather than a probabilistic multi-rater model. That is itself a publishable and useful negative result for the multi-rater literature.

Research Program

Data. We assemble a prostatectomy cohort with pre-surgical multi-parametric MRI and digitized whole-mount histology. We start from the public TCIA "Prostate-MRI" cases already used for registration benchmarking. Where data-sharing agreements allow, we add a mould-sliced cohort of the kind described in RAPSODI, whose 74 patients were cut with patient-specific 3D-printed slicing moulds.

Annotations. Five radiologists, blinded to histology, contour every index lesion on MRI, and two of them repeat the task weeks later so that within-reader inconsistency can be measured. Three pathologists independently outline tumor on each whole-mount slide. Following Validation of in vivo VERDICT fIC against matched histology, which used QuPath to produce objective epithelial cell-density maps, we also compute automated cell-density maps. These give a continuous histological reference that does not depend on any pathologist's pen.

Registration. Each slide is registered to MRI twice: once with RAPSODI and once with the weakly supervised deep network of Weakly Supervised Registration of Prostate MRI and Histopathology Images. RAPSODI is a 3D Slicer pipeline of rigid, affine, then free-form deformation. The weakly supervised network is VGG16-based and reports boundary Hausdorff distances of 4.38 mm internally and 5.01 mm externally. Both tools treat the pathologist outline as fixed truth; we do not. Registered anatomical landmarks give an empirical distribution of residual displacement $p(d)$, which RAPSODI reports as 2.88 ± 0.70 mm on average. Using two registration methods lets us check that conclusions do not depend on one tool's error profile.

Registration-aware reference. The histological reference at MRI location $x$ is defined as

$$ h(x) = \int p(d)\,\bar{T}(x+d)\,\mathrm{d}d, $$

where $\bar{T}$ is the per-pixel fraction of pathologists marking tumor after registration, $p(d)$ is the residual registration displacement density, and $h(x)$ is the probability that a correctly placed histological sample at $x$ would be called tumor. Any MRI-side prediction $q(x)$ is scored against $h$ with a proper scoring rule (the Brier score, and log-loss), together with reliability curves. This keeps a 3 mm registration slip from being charged to the radiologists. Variants that drop the displacement term, or replace $\bar{T}$ with a single pathologist, show how much each error source inflates apparent reader error.

Calibration model. The model is deliberately small. It is a monotone (isotonic) or low-dimensional logistic map from three inputs to $h$. The inputs are the agreement tier (0 to 5 readers), the signed distance to the majority contour, and the per-reader identity. Per-reader identity yields a histology-estimated, spatially varying "expertness" in the spirit of DiFF. DiFF learns rater weight maps by maximizing patient-level diagnostic accuracy, so its boundary is still judged by a proxy. We replace that proxy with local tissue.

Transfer to cases without histology. In routine care, histology and five readers are unavailable. We therefore apply the fitted calibration to the output of a learned multi-rater model (Pionono-style) trained on the radiologist contours. This tests whether histology calibration survives the move from real to simulated readers.

Baselines

Single radiologist and majority vote (naive reference). The weakest reference is one radiologist's contour. The next is the majority vote or STAPLE consensus of the five readers, which is what most pipelines would treat as ground truth. Both are scored against $h$ using the same folds. Any proposal must beat these by a clear Brier margin under a patient-level bootstrap. Voxels within a patient are strongly correlated, so every interval is resampled by patient.

Uniform margin expansion (strongest competitor). The consensus contour is dilated by a margin $m$. Separately, $q(x)$ is fitted as a function of signed distance to the consensus boundary alone. This is the strongest competitor because it directly encodes the underestimation that Gunashekar et al. found, and it has almost no parameters. Its margin is tuned by cross-validation on the same training patients, with the same folds and budget as our calibration model. Beating it means lower out-of-fold Brier and log-loss with a bootstrap interval excluding zero, plus higher Dice at each model's best threshold on held-out patients.

U-Net trained on radiologist contours. We run a network of the kind in Gunashekar et al., which reports a tumor-vs-histology Dice of 0.31 against a radiologist inter-observer Dice of 0.48–0.52. We retrain it on our cohort rather than borrowing numbers, since the cohort differs.

Uncalibrated multi-rater models. We compare a Pionono-style latent model and the Marin et al. ordered confidence map. These are trained on the five readers, and their raw probabilities are scored against $h$. The QUBIQ multi-rater challenge (QUBIQ) reports top Dice of 0.80–0.85, and Liu et al. report continuous Dice 0.5372 on QUBIQ brain growth. Both numbers measure agreement with human panels, not tissue. No reported result was found on Lacuna for any multi-rater segmentation model scored against histology.

Ablations. The key ablation removes the agreement tier from the calibration model and leaves distance only; this isolates whether disagreement matters. Further ablations remove the registration term $p(d)$ and replace the pathologist average with one pathologist.

Why not just…?

Why not just expand the consensus contour by a fixed margin? This is a real threat, and the study is built around it. If underestimation is roughly uniform around the lesion, a margin tuned on histology will match any agreement-aware model. The distance-only ablation settles it directly. If adding agreement tier and reader identity does not lower held-out Brier, we report that disagreement is uninformative about tumor extent and recommend the margin. We also test a middle case in which a lesion's overall disagreement predicts how large its margin should be.

Why not just train a segmenter directly on registered histology masks? Histology cohorts are small; the Gunashekar test set had 15 patients. Registration error of 3–5 mm is also comparable to the boundary effects in question, so a network trained on raw registered masks learns registration blur as tumor shape. Our calibration model has a handful of parameters that small cohorts can estimate, and it scores against a blur-aware reference. We include a histology-trained U-Net as an extra comparison, with the same folds.

Why not just use STAPLE or majority vote, or calibrate to diagnosis as DiFF does? A consensus of readers who share an underestimation bias inherits that bias, which is the specific failure Gunashekar et al. reported. DiFF-style weighting optimizes patient-level labels that say nothing about where a boundary lies. Both appear as baselines, so the claim is tested rather than assumed.

Why not just use deep ensembles or test-time dropout for uncertainty? Their spread reflects model uncertainty, not reader disagreement. Liu et al. report that Bayes dropout trails their variational model on continuous Dice (0.5083 vs 0.5372). More importantly, neither kind of uncertainty has been checked against tissue. We score ensemble variance against $h$ at matched compute to see whether it localizes tumor any better than reader disagreement.

Proposed Experiments and Decisive Checks

The first check is a tier-wise reliability curve. It plots registration-aware histological tumor probability against radiologist agreement tier, pooled over patients with bootstrap bands. It is computed with and without the registration term, and with one versus three pathologists, so the size of each error source is explicit.

The second check is the ring test. Within concentric rings at a fixed distance outside the consensus contour, we ask whether voxels marked by some readers contain more histological tumor, and higher QuPath cell density, than unmarked voxels at the same distance. Agreement labels are permuted within rings as a null. This is the decisive experiment. A significant within-ring effect supports the core claim; a null result favours the margin baseline.

The third check is out-of-fold comparison of the calibration model against every baseline above. Both registration methods are used, and a site-held-out split is added if the second cohort is available.

The fourth check applies the fitted calibration to a learned multi-rater model on held-out patients. It asks whether calibrated simulated disagreement retains the benefit found with real readers, which determines whether the result is usable without assembling five radiologists per case.

Every conclusion is reported with patient-level intervals and under both registration pipelines, so a finding that holds under only one is flagged as fragile.