Many clinical studies relate a number measured from an image to a clinical judgment, such as tumor volume against a malignancy rating or cancer area against a grade. The image measurement usually comes from a human tracing a boundary (a segmentation), and the judgment often comes from the same human. These studies are cross-sectional: they look at one moment in time, so their associations cannot tell forward causation from reverse causation or from a hidden common cause. We argue that, for imaging, inter-observer variability (the fact that different experts trace different boundaries on the same image) is not only noise that blurs these associations. When one reader supplies both quantities, that reader's private perception of a given case is a common cause of both. A radiologist who sees a nodule as larger also tends to see it as more malignant. This inflates the association and can make it look causal.
The fix comes from the structure of multi-rater datasets. Because several experts annotate the same case, we can pair one expert's measurement with a different expert's judgment. That pairing breaks the shared-reader path. Comparing the same-reader and cross-reader associations then gives a direct estimate of how much of a published imaging association is produced by the readers themselves. We propose to build this estimator and validate it where the truth is known. We then apply it to multi-rater benchmarks: LIDC-IDRI, Gleason 2019, and IMA++. We also test whether rater-shared error flips the arrows found by causal-discovery algorithms run on image-derived volumes.
Core Claim
Index cases by $i$ and readers by $j,k$. Let $m_{ij}$ be the measurement from reader $j$'s segmentation, for example nodule volume. Let $y_{ik}$ be reader $k$'s clinical label, for example a malignancy score. Write each as a true case-level value plus a reader-specific error, $m_{ij}=\mu_i+e_{ij}$ and $y_{ik}=\eta_i+u_{ik}$. The quantity a study wants is $\operatorname{Cov}(\mu_i,\eta_i)$. A same-reader analysis estimates that quantity plus $\operatorname{Cov}(e_{ij},u_{ij})$. A cross-reader analysis ($j\neq k$) removes that second term as long as different readers' errors on a case are uncorrelated beyond what the image itself explains. The shared-reader bias is therefore identified as
$$ \hat{B} = \widehat{\operatorname{Cov}}(m_{ij},y_{ij}) - \widehat{\operatorname{Cov}}(m_{ij},y_{ik})_{j\neq k}, $$
the difference between same-reader and cross-reader covariances, averaged over cases. A third reader's measurement $m_{ij'}$ can then serve as an instrument for $m_{ij}$. This is the classical errors-in-variables device: two noisy measurements of the same thing correct for attenuation. Applied here, it gives a slope free of both attenuation and shared-reader inflation. The claim is testable in two parts. First, $\hat{B}$ is substantially positive on real multi-rater data. Second, the cross-reader instrumental estimate recovers known effects in semi-synthetic data where same-reader, consensus, and mixed-effects estimates do not.
The direction-1 framing sets the limit. Cross-reader pairing removes reader-level confounding. It cannot remove image-level confounders that push all readers the same way, such as blur, stain drift, or partial-volume effects. We treat those as the residual unmeasured third variable in the sense of the cross-sectional literature, and we bound them using acquisition and tool metadata rather than pretending they are gone.
Relation to Prior Work
Three lines of work model inter-observer variability as an object in its own right, and they supply our tools for simulating readers. Pionono represents each rater as a Gaussian in a latent space whose mean captures systematic bias and whose variance captures inconsistency. The Latent Doctor Model learns doctor-specific styles from a support set and conditions disagreement on the image. Disentangling Human Error from Ground Truth estimates per-annotator pixel confusion matrices around a single latent truth. All three model how readers deviate in their masks. None of them asks how a reader's mask error co-varies with that same reader's diagnostic label, which is the quantity that contaminates downstream associations. We reuse Pionono-style rater models to generate realistic reader errors in our semi-synthetic studies.
DiFF is the closest work, and it motivates the proposal by being exposed to the problem it does not address. DiFF defines the "ground truth" mask as the fusion of raters that best helps a downstream diagnosis network, and reports AUC gains of up to 2.73% on glaucoma. When the masks and diagnostic labels share readers, a diagnosis-first objective can reward exactly the shared-reader component we isolate, so part of its gain could be circular. Our estimator offers a way to check this, which DiFF itself does not.
MCDRL marginalizes imaging confounders such as blur and mucus through a do-operation over a text-described confounder dictionary. It treats confounding as environmental and ignores annotators. We treat the annotator as the confounder and use the multi-rater design rather than a learned dictionary to remove it. IMA++ constrains our design. It shows that the annotation tool systematically shifts masks, and that 236 multi-annotated images have mean pairwise Dice below 0.5. Tool is therefore a second shared cause that cross-reader pairing removes only if readers differ in tools, and we must model it explicitly. Beyond Boundary Noise adds a second constraint: on lung CT, readers disagree about whether a nodule exists, not just where its edge is. Our measurement model must allow a reader-specific presence decision, not only additive volume error.
Finally, Longitudinal Causal Image Synthesis runs causal discovery (FCI, GES) over tabular variables that include brain volumes, and treats those volumes as exact. The cross-sectional direction-1 studies show the same habit: mediation models such as Xing et al. and Chen et al. assume an arrow direction that the data cannot verify. Neuroimaging associations such as cortical activation versus sleep quality inherit whatever error the image measurement carries. Our third experiment asks whether realistic segmentation error changes which arrows such algorithms draw.
Research Program
The first component is the estimator and its diagnostics. We will implement same-reader, cross-reader, and instrumental (reader $j'$ instrumenting reader $j$) estimates of the association between an image-derived measure and a reader-assigned label. Each will have case-level bootstrap intervals, and the bias term $\hat{B}$ will come with its own interval. Because readers do not annotate every case, we will use only cases with at least three readers for the instrumental estimate and report how many cases remain.
The measurement model will have three parts: an additive boundary error, a reader-specific presence decision following the presence-ambiguity finding, and a tool term following IMA++.
The second component is a semi-synthetic testbed with known truth. We will fit a Pionono-style rater model to real multi-rater masks, which gives realistic reader biases and inconsistencies. We will then generate true case-level volumes and labels with a chosen true effect. Reader labels will be produced through a controllable coupling $\rho$ between each reader's mask error and label error on the same case. Setting $\rho=0$ yields pure noise. Setting $\rho>0$ yields shared-reader confounding. A separate image-level term lets us test the method's stated blind spot.
The third component applies the estimator to real data and to causal discovery.
Baselines
The naive reference point is the same-reader regression: each reader's volume against that reader's label, pooled over readers. This is what a typical single-reader clinical study effectively computes. Beating it is necessary but not sufficient.
The consensus baseline computes the measurement from a STAPLE or majority-vote mask and the label from the mean or majority rating. Both sides are averaged over the same readers. This is the default in most imaging studies and is the strongest "standard practice" competitor. We will run it on exactly the cases and readers used by our estimator. Beating it means recovering the true semi-synthetic effect within its bootstrap interval when $\rho>0$, where the consensus estimate does not.
The mixed-effects baseline adds reader and tool random intercepts and slopes to the same-reader model. It is the statistician's strongest simple competitor, because it absorbs each reader's average tendency to draw large or rate high. It will receive the same covariates and the same data. We expect it to match our estimator when the shared error is purely a stable reader trait. We expect it to fail when the coupling is case-specific: a reader's perception of this particular case drives both quantities. The semi-synthetic testbed varies these two regimes separately.
The automated-measurement baseline replaces human volumes with those from a segmentation network trained on held-out cases. That network is the strongest available rater model, Pionono's consensus head. Its strength is that it removes the human from the measurement side entirely. It will be trained under the same compute budget and evaluated on the same cases.
For calibrating the rater simulator, we rely on the strongest reported multi-rater results in the file: Pionono's Cohen's kappa of 0.758 on Gleason 2019 (Pionono), the latent doctor model's entropy correlation of up to 0.728 with human panels on ISUP (LDM), and AmbiSSL's generalized energy distance of 0.1620 on LIDC-IDRI (AmbiSSL). For the actual question, the bias of image-derived associations under shared readers, there is no reported result on Lacuna for any of these datasets.
The key ablation keeps the instrumental estimator but forces the instrument to come from the same reader, for example a re-traced or perturbed mask. This isolates whether the gain comes from crossing readers or merely from using two measurements.
Why not just…?
Why not just average all readers on both sides? Averaging over $K$ readers shrinks the shared-reader covariance by about $1/K$, but it does not remove it. With the four to six readers typical of LIDC-IDRI or Gleason 2019, a meaningful fraction remains. The consensus baseline measures exactly this. If $\hat{B}$ turns out to be small relative to the true effect at realistic $K$, averaging is adequate, and we will report that as a finding rather than argue around it.
Why not just fit a mixed-effects model with reader terms? This is the real threat. If readers' shared errors are stable traits, for example "reader 3 always draws big and rates high," random slopes will capture most of the bias. Our argument rests on case-specific coupling, which no reader-level term can absorb. The semi-synthetic regime that separates trait coupling from case coupling settles which world we are in. On real data, the diagnostic is whether $\hat{B}$ stays positive after reader random effects are removed.
Why not just use an automated segmenter so the human only supplies the label? This removes the human from the measurement path, and we include it as a baseline. It has two problems. The network learns from the same readers' masks, so it reproduces their consensus perception. And IMA++ shows that tools introduce their own systematic shifts, so the automated volume carries a tool term that may correlate with how humans rate lesions the tool handles badly. It also discards the per-reader information that allows bias to be quantified at all.
Why not just model disagreement with an image-conditioned latent rater model, as the latent doctor model does? Such models describe the distribution of masks. They say nothing about the joint distribution of a reader's mask error and that reader's label, which is the object of interest. We use them as simulators, not as the estimator.
Proposed Experiments
The first experiment is the semi-synthetic sweep. We vary the coupling $\rho$, whether the coupling is a reader trait or case-specific, the number of readers per case, the presence-ambiguity rate, and the strength of an image-level confounder. For every estimator we report bias, interval coverage, and the rate of reaching the wrong sign. The decisive check: under case-specific coupling, the instrumental cross-reader estimate keeps near-nominal coverage while the same-reader, consensus, and mixed-effects estimates do not. Under a pure image-level confounder, all methods fail together, which is the honest limit.
The second experiment uses real multi-rater data. On LIDC-IDRI we relate each reader's nodule volume to that reader's malignancy score. On Gleason 2019 we relate each pathologist's high-grade area fraction to that pathologist's slide-level grade. For both we report $\hat{B}$ and the same-reader, cross-reader, and instrumental slopes. The prediction is that same-reader slopes exceed cross-reader ones, by an amount predicted by the measured within-case reader error covariance.
On IMA++ we use the tool and skill metadata. We restrict to readers who used more than one tool, so that tool and reader effects can be separated within reader. We then estimate how much of the disagreement tail is attributable to the tool rather than the person.
As a check on DiFF-style training, we retrain a diagnosis-first fusion on REFUGE-2 or ISIC using cross-reader labels instead of same-reader labels. If the reported AUC gain shrinks, part of it was shared-reader signal.
The third experiment tests causal discovery. We use the causal graph fitted to ADNI in Longitudinal Causal Image Synthesis as a generative ground truth. We add segmentation-like error to its volume variables, with magnitudes taken from the rater models fitted above, optionally coupled to a reader-rated clinical variable. We then rerun FCI and GES and count orientation errors. The decisive result is a monotone increase in flipped or spurious edges as reader-shared error grows. A second result would be that running discovery on cross-reader-paired variables restores the original orientations.
If this result holds, it means that cross-sectional imaging associations from single-reader studies need a reader-crossing correction before anyone draws their arrows.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Disentangling Human Error from Ground Truth in Segmentation of Medical Images
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation
- IMA++: ISIC Archive Multi-Annotator Dermoscopic Skin Lesion Segmentation Dataset
- Beyond Boundary Noise: Aggregated Aleatoric Uncertainty Fails to Capture Presence Ambiguity in 3D Lung Nodule Segmentation
- Longitudinal Causal Image Synthesis
- Association Between Sleep and Vertigo Severity in Benign Paroxysmal Positional Vertigo: Mediating Role of Psychological Factors
- Chain relationship between cumulative ecological risk and physical activity pattern on college students mediated by exercise atmosphere and self-control
- Relationship between cortical activation and sleep quality in cerebral small vessel disease patients: a functional near-infrared spectroscopy (fNIRS) study
- Annotation Ambiguity Aware Semi-Supervised Medical Image Segmentation