Who Drew the Contour? Correcting Rater Style in Imaging Biomarkers When Readers Are Not Randomly Assigned

Many clinical studies measure a tumor's size, shape or texture from a boundary drawn by hand, then ask whether that measurement predicts survival or malignancy. Different experts draw systematically different boundaries. Hospitals also rarely assign scans to readers at random. Senior radiologists tend to get the hard cases, and particular readers tend to cover particular scanners or clinics. The reader's habits therefore leak into the measurement, and the reader's caseload leaks into the outcome. The usual statistical fix is to add "which reader" as a variable in the outcome regression. That fix fails in exactly the situations that matter. It breaks when a reader's habit depends on what the lesion looks like, for example over-drawing only blurry margins. It breaks when there are many readers who each read only a few cases. It breaks when caseloads barely overlap, because then reader habit and patient severity cannot be told apart from the regression alone.

This proposal treats reader habit, or "rater style," as a systematic measurement error. That error can only be measured cleanly where several readers have outlined the same image. Comparing readers on the same image cancels the patient, so the remaining difference is pure style, whatever the hospital's assignment policy was. We would learn a model of each reader's style from a small double-read subset and use it to correct every measurement in a cohort where each case has only one reader. We would then test the corrected measurements with a semi-synthetic design. Public datasets in which every image was outlined by every reader let us simulate realistic, severity-driven reader assignment while knowing the true answer.

Core Claim

Our central claim is about identification, which here means being able to recover the true answer from the data. When reader assignment depends on patient severity, a reader's style is identifiable from overlapping annotations on shared images. It is not reliably identifiable from single-annotation data or from reader indicators in the outcome model. A style model trained on a modest double-read fraction, used to subtract each case's predicted style offset from its measured feature, should recover the true feature–outcome association with less bias and better confidence-interval coverage than reader fixed effects, consensus-free label models, or feature filtering. The advantage should grow as style becomes lesion-dependent, as readers multiply, and as caseloads separate.

Relation to Prior Work

The confounder-control side supplies the logic. Multivariable adjustment and its residualization form strip variance shared with a measured confounder. This is how obesity was shown to harm grafts independently of diabetes in Effect of the Obesity Epidemic on Kidney Transplantation. It is also how an apparently harmful prophylaxis turned out to reflect sicker patients receiving it in Venous thromboembolism prophylaxis practice and its association with outcomes in Australia and New Zealand burns patients. The second paper is the closest analogue to severity-driven reader assignment.

Beyond Simpson's Paradox constrains our design. It shows that pooled associations across heterogeneous actors can reverse once you stratify. That warns us that pooled radiomic associations across readers may be similarly misleading. It also warns that stratification alone needs overlap between strata to work.

The multi-rater segmentation literature supplies the machinery, but it stops before the outcome model. Three papers learn per-reader style:

None of the three exports style into a downstream association study. The single-label method of the third paper implicitly assumes readers are not matched to cases by difficulty. Our proposal tests that assumption and argues it breaks. MRNet and OMBA-UNet model readers with fixed expertise weights or one decoder per reader. Both need a closed reader pool and both stay inside the segmentation task.

DiFF is the only work that ties multi-rater fusion to a downstream task. It reports glaucoma AUC gains up to 2.73%. It does so by letting the diagnosis network choose which reader to trust. If reader choice is entangled with case mix, that loop can reward whichever reader happened to see the informative cases. Our proposal differs in kind. We do not optimize boundaries for prediction. We correct the measurement so that the estimated association is unbiased.

The radiomics papers motivate the stakes. Small boundary shifts destabilize the features in the thyroid nomogram study and the pituitary adenoma study. In the BraTS 2017 survival entry, shape features from contours are treated as error-free.

Research Program

We model a segmentation-derived feature for case $i$ read by reader $r$ as $F_{ir} = T_i + b_r(x_i) + e_{ir}$. Here $T_i$ is the reader-independent feature (for example lesion volume or cup-to-disc ratio), $b_r(x_i)$ is reader $r$'s systematic offset, which may depend on the image $x_i$, and $e_{ir}$ is that reader's own inconsistency.

The outcome depends on $T_i$ and on patient covariates. Reader assignment $r(i)$ may depend on severity. If $b_r$ were a constant, reader fixed effects would remove it. The research question is what happens when it is not constant.

Our estimator has three steps:

  1. Style model. Train a reader-conditioned segmentation model, using Pionono's architecture as the backbone, only on the double-read calibration subset. Add a within-image contrast loss: for every pair of readers on the same image, the model must reproduce the difference between their contours. This forces the style term to be learned from comparisons where patient severity is held fixed.
  2. Correction. For each single-read case, compute the predicted offset as the feature of the model's mask under the observed reader minus its feature under a reference reader marginal. Subtract that offset from the measured feature.
  3. Error propagation. Use Pionono's intra-reader variance to draw multiple corrected values per case. Combine the resulting outcome-model estimates as in multiple imputation, so that confidence intervals carry the correction's own uncertainty.

New readers enter through the support-set encoder of the Latent Doctor Model, which needs only a handful of their double-read cases.

The program then asks three things:

  • How large a double-read fraction is needed.
  • How the method fails when style really is constant, which is where fixed effects should tie us.
  • Whether it fails when the double-read subset itself is chosen non-randomly. Hospitals usually double-read only hard cases, so this last test matters most for practical use.

Baselines

Naive analysis. The do-nothing reference regresses the outcome on the single-reader measured feature plus patient covariates. It shows how much bias the reader assignment actually induces in each scenario.

Reader fixed effects. This is the strongest statistical competitor: reader indicators in the outcome regression, with a random-intercept mixed model as a variant. It is the standard confounder-control answer. It is exactly unbiased under constant additive style, so it is our main threat. It is fit to the same single-read cohort with the same covariates and no tuning advantage.

Consensus-label segmentation models. The strongest machine-learning competitors are the confusion-matrix model of Disentangling Human Error from Ground Truth and Pionono itself, used to predict a reader-free "gold" mask from which features are recomputed. Both get the same images, the same double-read subset and the same hyperparameter search budget as our model, with identical backbones. Pionono's reported unweighted kappa of 0.758 on Gleason 2019 serves as a check that our reimplementation is faithful before any downstream comparison. We have no reported result on Lacuna for any of these methods on downstream association bias.

Feature filtering. Discard features with low inter-reader agreement on the double-read subset and analyze the rest.

Oracle. Features averaged over all readers on every image give a target that is unavailable in practice, since it requires reading every case multiple times.

Ablations. The key ablation removes the within-image contrast loss and trains the style model on single-read data only. If that version matches the full method, our identification claim is wrong. A second ablation feeds Pionono's latent reader mean as a covariate into the outcome model, which is the obvious "export the latent" idea.

Beating the baselines means lower absolute bias in the feature coefficient relative to the oracle, and 95% interval coverage near nominal, across the assignment scenarios where fixed effects is not exact. It also means no meaningful loss where fixed effects is exact.

Why not just…?

Why not just add reader indicators to the regression? Under constant additive style this is consistent, and we expect it to tie our method there. We will show that regime explicitly rather than hide it. It breaks in three ways:

  • Style that depends on the image leaves residual confounding inside each reader's cases.
  • Separated caseloads give each reader a narrow severity band, which inflates variance and leaves the reader-versus-severity split resting on extrapolation.
  • Many sparse or unseen readers make indicators unestimable.

The experiments vary each knob separately. If fixed effects matches us at realistic settings, we will report that the simple method suffices.

Why not just double-read everything, or use STAPLE consensus? Full multi-reading removes the problem but costs readers' time on every case. Collapsing readers into a consensus mask also discards ambiguity, as the label-distribution results of the Latent Doctor Model show. Our real question is the cost curve: how small a double-read fraction recovers most of the oracle's accuracy. The full-consensus oracle is in the comparison precisely to anchor that curve.

Why not just drop features that readers disagree on? Filtering removes unstable features but leaves the surviving ones still correlated with reader assignment. It also throws away contour-sensitive signals such as volume and compactness, which are the core of shape-based survival models like the BraTS 2017 entry. That model reached a test Spearman correlation of only 0.496. Our filtering baseline measures both residual bias and lost signal.

Why not just use the latent reader vector as a covariate? Pionono's latent means are learned across images. Under severity-driven assignment they can absorb caseload as well as habit, so adjusting for them may remove true signal. The second ablation tests this directly.

Proposed Experiments

The decisive experiment is semi-synthetic. It uses datasets where every image was outlined by a fixed reader panel: RIGA (750 fundus images, 6 glaucoma specialists), Gleason 2019 (244 cores, 6 pathologists), and the QUBIQ tasks (3 to 7 readers). We will first confirm that reader identities are consistent across images in each. For LIDC-IDRI we must verify that reader labels link across scans before using it this way.

For each dataset:

  1. Define the true feature as the all-reader mean of a clinically meaningful measurement: cup-to-disc ratio, tumor-area fraction, or organ volume.
  2. Generate outcomes from that feature plus a covariate, with a known coefficient.
  3. Build single-read cohorts by assigning one reader per case under four policies: random, severity-driven, scanner or site-driven, and near-disjoint caseloads.
  4. Separately inject controlled lesion-dependent style by eroding or dilating contours more on low-contrast boundaries. This follows the synthetic-annotator protocols of Tanno et al. and OMBA-UNet.

Because every reader's contour exists for every image, the counterfactual is fully known.

We will report coefficient bias, interval coverage, and out-of-sample discrimination when the deployment assignment policy differs from the training policy. We will sweep the double-read fraction from 5% to 50%, drawn both randomly and from the hardest cases.

The real-outcome check uses LIDC-IDRI malignancy. Here the outcome for each nodule is the mean score of the readers who did not draw the contour being analyzed. This leave-one-reader-out construction avoids the same-reader contamination that the vessel-distance analysis of the LIDC-IDRI nodule context study did not address. Where available, we also check against the pathology-verified subset.

The result that would sink the proposal is the no-contrast ablation, or reader fixed effects, matching the full method under severity-driven assignment with lesion-dependent style. The result that would establish it is a widening gap in bias and coverage as style dependence and caseload separation increase, with most of the gain achieved by a double-read fraction of 10–20%.