When several radiologists score the same lung nodule for malignancy, their disagreement is usually treated as one of two things. It is either noise to be averaged away, or a property of the image and of each radiologist's personal style, to be modeled. This proposal tests a third explanation. Part of what radiologists disagree about, and part of what they systematically over-call, may be driven by how the scan was acquired rather than by the nodule. Thick slices, sharp reconstruction filters and particular scanner vendors can make a benign nodule look ill-defined, and a cautious reader then scores it higher. If that is true, a model trained on these scores inherits a confounder: a variable (here, acquisition settings) that shapes the label without being part of the disease. Existing confounder-removal methods cannot remove it, because they assume the label itself is clean.
We propose a rater model in which each radiologist's score has two inputs. One is a learned representation of the nodule, which we force to carry no information about acquisition. The other is an explicit pathway by which acquisition settings shift each rater's scoring. The second pathway absorbs the "annotation disparity", meaning bias introduced by the labeling process itself. That leaves the representation free to track histology. We judge success against biopsy-proven ground truth, not against radiologist consensus.
Core Claim
The causal analysis of dataset bias in A causal perspective on dataset bias in machine learning for medical imaging names annotation disparities, a causal arrow from a non-disease attribute into the labeling process, as a distinct bias family. That work is purely conceptual and gives no algorithm or experiment. Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction shows the practical damage of rater bias in the LIDC-IDRI lung CT dataset. Radiologists' "safety-first" scoring leaves consensus-trained models with sensitivity 0.97 and specificity 0.23. The authors repair this by propagating biopsy labels from 180 pathology-proven nodules (SCH-LND) through a learned similarity space. They do not ask what makes radiologists over-call, and they explicitly do not control acquisition differences between the anchor and target datasets.
Our claim is that a measurable share of malignancy over-calling and inter-rater disagreement in LIDC-IDRI is explained by acquisition covariates, after controlling for nodule size. We further claim that modeling this pathway explicitly in the rater-generation process yields a representation that predicts biopsy truth better than consensus training, soft-label training, or confounder removal applied to consensus labels. The novelty does not lie in confounder removal under multi-rater labels as such. It lies in recognizing that conditioning on rater labels, as Training confounder-free deep learning models for medical applications does, preserves exactly the confounding we want to remove when the labels are themselves confounded.
Research Program
The program has two stages. The first is cheap, and it can kill the project.
Stage one: is there annotation disparity? We extract, for every LIDC-IDRI nodule, the acquisition covariates available in the scan headers: slice thickness, reconstruction kernel, manufacturer and tube current. We also extract nodule diameter and location from the annotations. Whether these fields are complete enough across LIDC-IDRI must be verified first. We then fit a mixed-effects ordinal regression, meaning a regression for ordered 1–5 scores with random effects per radiologist and per nodule. It regresses each individual rating on these covariates. A second model regresses the per-nodule spread of ratings on the same covariates. If acquisition terms explain nothing beyond size, we stop and report the negative result, because the method below would then reduce to existing rater-style models.
Stage two: the model. Let $x_i$ be the CT patch for nodule $i$, $c_i$ its acquisition covariates, $h(x_i)$ a representation from a 3D convolutional encoder, and $r_{ij}$ the score of rater $j$. We model
$$ P(r_{ij} \le k) = \sigma\big(\theta_{jk} - f(h(x_i)) - a_j^\top c_i\big), $$
where $\sigma$ is the logistic function, $\theta_{jk}$ are rater-specific cut points (a rater's personal strictness), $f$ maps the representation to a malignancy score, and $a_j$ is how strongly rater $j$'s scoring shifts with acquisition.
Without a constraint, the encoder could simply read slice thickness off the image, and the acquisition pathway would be unidentifiable. We therefore add an adversary in the style of CF-Net that tries to predict $c_i$ from $h(x_i)$, and train the encoder to defeat it. Marginal independence would erase real signal if acquisition correlated with malignancy, which is the over-correction failure noted for fairness methods. To guard against this, we apply the adversary conditionally on pathology label wherever pathology is known. For a portion of SCH-LND held out by cross-validation, this is $h \perp c \mid \text{pathology}$, following CF-Net's conditional game. On LIDC-IDRI the adversary is applied marginally. We justify the marginal constraint only after checking, on the anchor set, that acquisition covariates are near-independent of histology.
At inference we predict from $f(h(x))$ alone, with the acquisition pathway set to a reference protocol.
A secondary setting is VinDr-CXR (VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations). There, 15,000 training images were each labeled by three radiologists, and images come from two hospitals, so site and device can serve as the covariate. It has no pathology ground truth, so we would use it only to test whether the rater-level acquisition effects replicate. That requires rater identifiers in the release, which we must confirm.
Baselines
The naive reference point is a 3D ResNet-18 trained on the LIDC-IDRI consensus score, binarized as in the relabeling paper. On that paper's evaluation it reports sensitivity 0.97 and specificity 0.23. Every method uses the same encoder, the same nodule crops, the same augmentation and the same hyperparameter search budget, here twenty trials per method tuned on an LIDC-IDRI validation split. All methods are evaluated on the identical biopsy-labeled folds.
The strongest existing method is the biopsy-anchored relabeling of Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction, which reports sensitivity 0.67 and specificity 0.66. It is the strongest competitor because it already uses the same biopsy anchors and directly targets rater over-calling. We re-implement it on our folds so that it sees exactly the same anchor nodules as our conditional adversary. Beating it means higher area under the ROC curve against pathology at matched anchor usage, with non-overlapping bootstrap intervals. It also means a specificity gain at sensitivity fixed to its operating point. We also test the combination of their relabeling with our encoder, since the two ideas are complementary rather than exclusive.
The multi-rater competitor is a rater-style model without covariates, following The Latent Doctor Model for Modeling Inter-Observer Variability, which reports KL divergence 0.461 to the expert panel on Gleason2019. In our model this corresponds to dropping the $a_j^\top c_i$ term but keeping rater cut points. This is the key ablation, because it isolates whether the acquisition pathway adds anything beyond rater style. A second ablation keeps the acquisition pathway but removes the adversary. This tests whether the explicit pathway alone suffices, or whether the encoder simply re-encodes acquisition. The third baseline is CF-Net trained on consensus labels, which reports 74.1% balanced accuracy versus 71.6% for a plain network on HIV MRI. It tests whether ordinary confounder removal already solves the problem.
Why not just…?
Why not just residualize the rater scores on acquisition covariates and train on the residuals? This is the most serious threat. If acquisition enters the scores roughly additively, regressing each rating on covariates and training on corrected soft labels could match our model at a fraction of the complexity. The confounder-free paper argues residualization fails for high-dimensional image features. Here, though, we would residualize the labels, so that argument does not settle the question. We therefore include label residualization, using the stage-one ordinal regression to produce corrected scores, as a full baseline. If it matches the end-to-end model on biopsy AUC, we will report the two-stage pipeline as the contribution. The adversary would then be justified only if residualized training still lets the encoder exploit acquisition cues in the image, which we measure with a post-hoc probe predicting $c$ from the representation.
Why not just apply CF-Net to the soft multi-rater label? Conditioning on rater labels, whether consensus or the full distribution, keeps any acquisition dependence those labels contain. If radiologists over-call thick-slice nodules, the critic is told that thick-slice nodules are "more malignant" and leaves that association intact. The CF-Net-on-consensus baseline and a soft-label variant will show this directly. The acquisition probe should still recover slice thickness from their representations, and their specificity should be lowest on thick-slice biopsy nodules.
Why not just rely on rater-style models, since per-rater thresholds already capture caution? Rater cut points capture how strict each radiologist is on average. They cannot capture caution that varies with the scan, which is the hypothesis here. The covariate-free ablation tests this. If stage one finds per-rater acquisition slopes indistinguishable from zero, rater-style models are sufficient and our proposal adds nothing.
Why not just harmonize scans, for example by resampling to common slice thickness? Resampling cannot restore detail lost in thick acquisitions, and it leaves the radiologists' biased scores untouched. We will nonetheless include resampled inputs as a preprocessing variant for all methods, so that any gain is not explained by harmonization alone.
Proposed Experiments and Decisive Checks
The first decisive check is stage one. We need a statistically reliable acquisition effect on individual ratings and on rating spread, after nodule size and rater random effects. Absent that, the project ends with a published negative result about LIDC-IDRI annotation disparity.
The second check is biopsy-truth performance on SCH-LND under five-fold cross-validation. We report ROC AUC, sensitivity and specificity, and expected calibration error. Results are stratified by slice-thickness tertile, since the model's claimed advantage should concentrate where acquisition is worst. A uniform gain across strata would suggest the benefit comes from something else, such as regularization.
The third check is internal consistency. The learned per-rater slopes $a_j$ should agree in sign with the stage-one regression. The acquisition probe on our representation should be near chance, while probes on all baselines should not be.
Finally, the presence-ambiguity signal found in Beyond Boundary Noise: Aggregated Aleatoric Uncertainty Fails to Capture Presence Ambiguity in 3D Lung Nodule Segmentation gives a test of over-correction. That paper's ambiguity head reaches AUROC 0.74 on LIDC-IDRI. We will fit the same post-hoc head on our frozen representation. A large drop would mean the adversary erased real ambiguity along with acquisition. A small drop, together with lower correlation between ambiguity and slice thickness, would mean some of the "ambiguity" radiologists express was acquisition all along.
References
- A causal perspective on dataset bias in machine learning for medical imaging
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
- Training confounder-free deep learning models for medical applications
- VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Beyond Boundary Noise: Aggregated Aleatoric Uncertainty Fails to Capture Presence Ambiguity in 3D Lung Nodule Segmentation