When the Reader Is a Confounder: Correcting Cross-Sectional Estimates for Ambiguity-Dependent Diagnostic Disagreement

Many medical studies take a single snapshot of a population. They measure exposures such as age or sex and a diagnosis at the same time, then report how common the disease is (its prevalence) and how strongly it tracks each exposure, usually as an odds ratio. This design is called a cross-sectional study. Increasingly, the diagnosis in such studies comes from a radiologist's read or from an AI model trained on radiologists' reads, and the literature treats that label as the truth. But experts disagree on the same image, and they disagree most on ambiguous images. If ambiguity itself varies with the exposure, the label errors differ between exposure groups. Older patients' chest films, for example, are harder to call for cardiomegaly (an enlarged heart). Epidemiologists call this differential misclassification. It can inflate, shrink, or reverse an odds ratio, and no amount of extra sample size fixes it.

We propose to estimate, from images that several readers labeled independently, how each reader's error rates change with image difficulty, and to use that estimate to correct prevalence and odds-ratio estimates. The correction applies whether the cohort was labeled by one reader per image or by an AI model that inherited one reader's habits. The test bed is VinDr-CXR, a public chest X-ray set where every training image has three independent radiologist reads and every test image has a five-reader consensus. The decisive question is whether conditioning reader error on image difficulty removes odds-ratio bias that both consensus labels and classical rater-specific error correction leave behind.

Core Claim

Disagreement among experts is not only label noise for training classifiers. In a cross-sectional study it is a source of bias in the scientific estimate, and the bias has a specific, testable structure. Write $Y$ for the unobserved true finding, $Z$ for an exposure, $\tilde Y_r$ for reader $r$'s label, and $d(x)$ for a scalar difficulty score of image $x$. We model each reader's chance of calling a finding positive as

$$\operatorname{logit} P(\tilde Y_r = 1 \mid Y = y, x) = \alpha_{r,y} + \beta_{r,y}\, d(x).$$

Here $\alpha_{r,y}$ is reader $r$'s baseline threshold for images whose true status is $y$, and $\beta_{r,y}$ says how that threshold drifts as images get harder. Whenever $d(x)$ is correlated with $Z$ and some $\beta_{r,y} \neq 0$, misclassification is differential in $Z$. A correction that ignores the image cannot remove the resulting bias. This includes the classical latent-class approach, which gives each rater one fixed confusion matrix. We claim that with three or more independent reads per image, the parameters above and the image-conditioned prior $p(Y \mid x)$ can be estimated jointly. The corrected association, computed from posterior probabilities of $Y$, is then close to what a large expert panel would report.

Positioning Against Prior Work

Several papers motivate the gap. The Impact of an Inter-rater Bias on Neural Network Training trained identical segmentation networks on different raters. It showed that the networks reproduce and even sharpen each rater's style: a classifier separated the networks' outputs by source better (hit rate up to 0.93) than it separated the raw annotations. A network trained on the junior rater underestimated multiple sclerosis lesion load. That work stops at volume and overlap metrics, and we take its result as the reason AI-phenotyped cohorts inherit reader thresholds at population scale. Causal Representation Learning with Observational Grouping for CXR Classification treats CheXpert and MIMIC-CXR as confounded observational data and enforces invariance across sex, race, and view groups. It still takes the labels as ground truth, which is exactly the assumption we drop. Difficulty-aware Glaucoma Classification with Multi-Rater Consensus Modeling shows that experts split along sensitive and conservative thresholds and that disagreement concentrates on hard cases. That supports the difficulty term $d(x)$, but the paper uses it only to reweight training.

Other papers supply method pieces. The Latent Doctor Model for Modeling Inter-Observer Variability encodes each doctor's style from a support set of their labels and predicts the full distribution of expert opinions. We borrow the idea of rater-specific parameters learned from that rater's labels. We differ in target: we estimate a latent truth and propagate it into population estimates, not into a panel-entropy match. Leveraging Inter-Rater Agreement for Classification in the Presence of Noisy Labels recovers a label-noise matrix from agreement statistics alone, without gold-standard samples. It assumes homogeneous, symmetric, image-independent noise, which we treat as the weakest correction baseline. More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel trains one network per pathologist and averages them. That is a strong labeler, but its calibration targets the panel's mixture of opinions, not the truth, and it excluded the ambiguous "indefinite for dysplasia" class.

Research Program

The work has three parts.

The first part is the error model. A shared image encoder outputs both $p(Y \mid x)$ and $d(x)$ for each finding. Reader-specific parameters $\alpha_{r,y}, \beta_{r,y}$ are fit jointly by expectation-maximization over the three VinDr-CXR training reads. Reader identifiers are needed for this. We understand the VinDr-CXR release attaches a reader ID to each annotation. If that proves wrong, the reader-specific terms collapse to pooled terms, and rater-level analysis moves to the Dutch Esophageal Pathology Panel (DEPP) used by the virtual-panel work, where about 11 pathologists graded each case.

The second part is the estimator. For a target odds ratio, the corrected estimator fits the logistic regression of $Y$ on $Z$ using posterior weights $p(Y \mid x, \tilde Y)$. Uncertainty is propagated by bootstrap over images and over the error-model fit. When only a single reader or an AI labeler is available at deployment, its error model is estimated on the multi-read calibration subset and then inverted.

The third part is an audit quantity: the share of an apparent association that is explained by reader identity and difficulty rather than by $Y$. A study can report this alongside its odds ratio.

Baselines

The naive reference is the analysis most cross-sectional studies would actually run: each image is labeled by whichever single radiologist read it, and the odds ratio is computed directly. The common improvement is the majority vote of the three reads.

The strongest labeler-side competitor is a virtual expert panel in the style of More is more: one classifier per radiologist, averaged, with the averaged probabilities used as soft outcomes in the regression. On DEPP that paper reports an average F1 of 78.2% for the Full Panel against 72.9% for the consensus model, and an expected calibration error (ECE) of 0.050 against 0.183.

The strongest correction-side competitor is the agreement-based noise-matrix recovery of Leveraging Inter-Rater Agreement, which reports 79.4% test accuracy against 74.0% for majority voting on CIFAR10-N. We apply its recovered matrix to correct prevalence and the odds ratio.

The key ablation is our own model with $\beta_{r,y}=0$. This is a rater-specific, image-independent confusion model, essentially the classical latent-class correction, and it isolates the difficulty term.

No reported result was found on Lacuna for prevalence or odds-ratio recovery on VinDr-CXR or DEPP, so all baselines are run by us. All methods use the same backbone, the same splits, the same exposures, and the same hyperparameter search budget. Beating the strongest competitor means two things on the held-out reference: lower absolute log-odds-ratio bias, and 95% interval coverage near nominal where the competitors fall short, across findings where difficulty correlates with the exposure.

Why not just…?

Why not just use majority-vote or consensus labels? Majority voting averages out reader-to-reader noise, but not shared, ambiguity-driven error. If all three readers tend to over-call cardiomegaly on difficult films from older patients, the vote does too. The virtual-panel work also shows that consensus training leaves models badly miscalibrated (ECE 0.183). Our experiments measure the remaining odds-ratio bias directly.

Why not just use classical latent-class correction with rater-specific confusion matrices? This is the real threat. If difficulty is uncorrelated with the exposure, or readers' thresholds do not drift with difficulty, it should match our method at lower complexity. The $\beta=0$ ablation settles the question finding by finding. We will report the measured correlation between $d(x)$ and each exposure, and we will state where the simpler model suffices. A null result there would be a useful finding about when the simple correction is enough.

Why not just use a calibrated multi-rater AI labeler such as a virtual panel? A well-calibrated panel predicts what readers would say, not what is true. Its soft labels therefore carry the readers' collective, difficulty-linked bias straight into the regression, and the inter-rater bias paper suggests trained networks sharpen rather than dilute such bias. The panel baseline run under identical conditions tests this.

Why not just collect more reads? More reads per image reduce random disagreement but not shared ambiguity bias. More reads per image are also exactly what large screening studies cannot afford. Our method needs multiple reads only on a calibration subset, and an experiment that varies the size of that subset quantifies the cost.

Proposed Experiments

The primary experiment uses VinDr-CXR. We target cardiomegaly and aortic enlargement, the two most common abnormalities at 12.1% and 15.6% training prevalence, against age and sex taken from image metadata.

The reference estimate is computed on the 3,000 test images using the five-reader consensus. For held-out training images we simulate realistic deployment: each image keeps one read, and reader assignment is deliberately correlated with patient age, mimicking site- or shift-level confounding. Separately, an AI labeler trained on a single reader phenotypes the cohort. Every estimator is then compared with the reference on prevalence error, log-odds-ratio bias, and interval coverage.

A semi-synthetic version plants known reader thresholds and a known exposure–difficulty correlation onto the real images. This gives exact ground truth and lets us map bias as a function of that correlation.

The DEPP experiment subsamples one to three of roughly eleven pathologists per case and checks recovery of the full-panel grade distribution.

Three results would decisively support the claim: our method approaches the five-reader reference where the $\beta=0$ ablation does not; the gap grows with the correlation between difficulty and exposure; and coverage holds with a calibration subset of a few thousand triple-read images. If the ablation matches us everywhere, the conclusion is that rater-specific but image-independent correction is enough for chest X-ray epidemiology.