Separating Scanner Shift from Reader Shift in Clinical-Radiomic Models

Many hospital prediction models combine a "radiomic score" with ordinary clinical variables. The radiomic score is a number computed from hundreds of texture and shape measurements inside a tumor outline that a radiologist draws on a CT scan; the clinical variables are things like age, stage, and lab values. These models often score well in the hospital that built them and then lose accuracy elsewhere. The usual explanation is that other hospitals use different scanners. That explanation leaves out a confound. At every new hospital, a different group of radiologists draws the tumor outlines, and outline disagreement alone leaves about half of manually derived radiomic features below acceptable reproducibility (Balagurunathan et al.). Live deployment adds a further change: the outline comes from whoever is on duty, or from an automatic segmenter that a busy clinician corrects. Standard multicenter studies cannot tell how much of the lost accuracy comes from the scanner and how much from the reader.

We propose a crossed reader-by-center study in which readers from every hospital outline scans from every hospital. This design splits the variation in the radiomic score, and in the final fused prediction, into an acquisition part, a reader part, and their interaction. We then train the fusion model to give stable predictions across real readers' outlines. Finally, we test prospectively, with consecutive patients, whether the split measured on retrospective data predicts the accuracy drop seen in live use for each outline source (local manual, automatic, automatic then edited). The application is predicting major pathological response after neoadjuvant chemoimmunotherapy in non-small cell lung cancer (NSCLC). For this task the best-known clinical-radiomic model comes from a single, 91.9% male center (Wang et al.).

Core Claim

The claim has two parts. First, the accuracy a clinical-radiomic model loses between internal testing and external or prospective use is partly caused by a change in who or what produces the tumor outline, which we call contour-source shift. In multicenter data this effect is confounded with scanner shift. It can be measured separately only when readers and centers are crossed. Second, the reader part can be reduced by training on several real outlines per patient and penalizing how much the fused prediction varies across them. Synthetic outline perturbations or feature filtering do not achieve the same reduction, because real readers disagree in structured ways rather than at random. They differ, for example, on whether to include spiculations, necrosis, or adjacent atelectasis, and eroding or dilating a mask does not imitate those choices.

The literature gives this claim both motivation and a direct threat. Balagurunathan et al. found feature reproducibility, measured by the intra-class correlation coefficient (ICC, the share of feature variance due to true differences between patients), of 0.77 for five physicians' manual contours versus 0.85 for semiautomatic contours. Only 52% of manual features reached ICC ≥ 0.8. Ferrante et al. found the opposite at the level of the downstream model: survival classifiers built on automatic nnU-Net contours reached accuracy 0.78, against 0.65–0.73 for manual contours, with no significant difference. They concluded that contour source hardly matters downstream. Our design can come down on Ferrante's side. If the reader component of the fused prediction turns out to be negligible, that is a real result, and it would favor the simple "just use nnU-Net" alternative over our method.

Research Program

The first component is the crossed reader study. Across four or more centers with different scanner vendors, we assemble retrospective cohorts of operable NSCLC patients treated with neoadjuvant chemoimmunotherapy, all with pathological response as ground truth. Each center contributes at least two thoracic radiologists. Every reader outlines a stratified subset of cases from every center, including their own, using the same viewer with no consensus step. This partially crossed design keeps each reader's workload bounded. Each patient therefore has several real outlines, and each outline is labeled with its reader, the reader's home center, and the scan's center. Readers also record RECIST measurements (the standard tumor-diameter response criteria), since this clinical input is itself reader-dependent.

The second component is the variance decomposition. For patient $p$ scanned at center $c(p)$ and outlined by reader $r$ from home center $h(r)$, we fit a mixed model to the radiomic score $s_{p,r}$, and separately to the fused predicted log-odds:

$$ s_{p,r} = \mu + u_p + a_{c(p)} + b_r + g_{h(r)} + (ag)_{c(p),h(r)} + \varepsilon_{p,r}. $$

Here $\mu$ is the overall mean and $u_p$ is the true patient effect. $a_{c(p)}$ is the scan-center (acquisition) effect, $b_r$ is the individual reader effect, and $g_{h(r)}$ is the effect of the reader's home-center habits. The term $(ag)$ captures readers who outline unfamiliar scanners differently, and $\varepsilon$ is the residual. In an ordinary multicenter study $c(p)=h(r)$ always holds, so $a$ and $g$ cannot be estimated separately. Crossing readers with centers is what makes them estimable.

The third component is a rater-invariant fusion model. We keep the standard late-fusion structure: LASSO selection of radiomic features into a score, followed by logistic regression with clinical covariates and RECIST. This keeps the model comparable to published nomograms. The change is in the training objective. For fused prediction $f(x_p^{(r)}, z_p)$, with outline-derived features $x_p^{(r)}$ and clinical covariates $z_p$, we minimize

$$ \mathcal{L} = \frac{1}{N}\sum_{p}\frac{1}{|R_p|}\sum_{r\in R_p}\ell\big(y_p, f(x_p^{(r)}, z_p)\big) + \lambda\,\frac{1}{N}\sum_p \operatorname{Var}_{r\in R_p}\big[f(x_p^{(r)}, z_p)\big]. $$

$\ell$ is cross-entropy against pathological response $y_p$, $R_p$ is the set of readers who outlined patient $p$, and $\lambda$ trades accuracy against stability across readers, tuned on held-out centers. The penalty acts on the fused prediction rather than on individual features. A feature that is unstable across readers can therefore be kept if the clinical covariates or other features cancel its instability, which ICC filtering cannot allow. This adapts the "diagnosis-first" idea of Wu et al., who reweight multi-rater segmentations to maximize a downstream image classifier. We move it to a multimodal model with tabular covariates. We also aim at stability across readers rather than at a single learned consensus outline, because no single consensus exists in deployment.

The fourth component is prospective validation. At each center, consecutive eligible patients are enrolled before surgery over roughly twelve months. Each patient's scan is processed three ways. The first is a manual outline by the on-duty local reader. The second is an automatic outline from an nnU-Net model trained on pooled multicenter data, following Holzschuh et al., who reached a leave-one-center-out median Dice of 0.74. The third is that automatic outline after the local reader edits it, with the editing time logged. All predictions are recorded before pathology is available. This addresses the gap identified across the prospective literature: Shen et al. validated a multimodal model prospectively, but the radiologist entered it only as a fixed categorical score, and no prospective study has varied how the imaging input is produced.

Baselines

The naive reference point is a clinical-only logistic model using the same covariates and RECIST. On this task, Wang et al. report a clinical-only AUC of 0.66. AUC (area under the ROC curve) measures how well a score ranks responders above non-responders. If our fused model cannot clearly beat this in external centers, adding radiomics is not justified.

The strongest existing method is the standard single-contour clinical-radiomic nomogram of Wang et al.. It combines a seven-feature CT radiomic score with RECIST and reports a validation AUC of 0.80, against 0.60 for radiomics alone. That figure comes from a single-center internal split, so we expect it to fall under leave-one-center-out evaluation. We re-implement the nomogram on our data with one outline per patient, taken from that patient's home-center reader to mimic how such models are normally built. It gets the same feature extractor, the same LASSO search grid, and the same number of tuning trials as our model. Two further competitors are the same nomogram built on pooled-multicenter nnU-Net contours, following Ferrante et al., and an ICC-filtered nomogram that keeps only features with cross-reader ICC ≥ 0.8, following Aerts et al.. The latter is the classic stability strategy behind a signature that held concordance of 0.65–0.69 across seven external cohorts. We also include a multimodal adaptation of DiFF (Wu et al.), in which reader outlines are fused with weights optimized for downstream accuracy before feature extraction.

Two ablations isolate the key ingredient, real-reader variance control. The first trains on all real outlines as augmentation with $\lambda=0$. The second keeps the variance penalty but replaces real outlines with synthetic erode/dilate/boundary-jitter perturbations calibrated to match the average Dice between readers. All models are evaluated leave-one-center-out and prospectively. Beating the strongest baseline means two things together. The first is a higher pooled leave-one-center-out AUC than both the Wang-style nomogram and the nnU-Net nomogram, with bootstrap confidence intervals excluding zero difference. The second is a smaller prospective AUC drop between contour sources, together with at least a halving of the reader component's share of variance in the fused prediction, without losing more than 0.02 AUC relative to the best unconstrained model.

Why not just…?

Why not just replace human outlines with nnU-Net? Removing the reader from the loop is the most serious threat. Ferrante et al. found no downstream loss with automatic contours. But in prospective use, clinicians edit automatic outlines, and those edits bring the reader back in. In addition, Holzschuh et al. show that nnU-Net's own accuracy depends on which centers it was trained on, which makes it a center-dependent reader of its own. The prospective arm measures all three contour sources directly. If the automatic-unedited source gives the best and most stable prospective AUC and the decomposition shows little reader variance, we will report that the simpler approach wins.

Why not just discard unstable features by ICC? Aerts et al. show that filtering yields signatures that transfer across cohorts. Under manual contours, however, filtering removes about half the feature set (Balagurunathan et al.). It also judges each feature on its own, when what matters is whether the fused prediction is stable. The ICC-filtered baseline tests this head to head. If it matches our model on both accuracy and stability, our penalty adds nothing.

Why not just simulate reader variability with mask perturbations? This needs no multi-reader data and would make the expensive crossed study unnecessary. Our hypothesis is that real disagreement is structured and center-dependent, which is exactly the $(ag)$ term. The synthetic-perturbation ablation, together with a direct comparison of the decomposition from real versus synthetic outlines, settles the question. If synthetic outlines reproduce the reader variance and the downstream benefit, that is a useful negative result, and future studies could skip crossed reading.

Why not just pool more centers or harmonize scanner differences? Multicentric pooling clearly helps (Holzschuh et al.), and we use it throughout. Pooling and scanner harmonization, however, act only on the acquisition term $a$, and in uncrossed data that term silently absorbs reader habits. The decomposition shows how much of the cross-center drop harmonization could possibly fix.

Proposed Experiments and Decisive Checks

The first decisive check is whether the reader component is material. We report the variance shares of $a$, $b+g$, and $(ag)$ for the radiomic score and for the fused prediction, with confidence intervals from the mixed model. A reader share near zero refutes the core claim and favors the nnU-Net baseline.

The second is retrospective leave-one-center-out evaluation of all baselines and ablations. Metrics are AUC, calibration slope, and decision-curve net benefit (the benefit of acting on the model's predictions at clinically relevant thresholds), plus the proportion of patients whose predicted response class flips across readers.

The third is a pre-registered prospective prediction. From the retrospective decomposition we predict, for each contour source, the AUC drop and flip rate expected in live use, and we register those numbers before enrollment ends. The study succeeds as science if the predictions fall within their intervals, whichever model wins. It succeeds as a method if the rater-invariant model shows a smaller prospective drop and flip rate than the Wang-style and nnU-Net nomograms. We also log edit time for the automatic-then-edited source as a workflow measure, in the spirit of Kottlors et al..

As a secondary sanity check of the penalty on public multi-rater data without clinical variables, we compare it with DiFF on TNMIX and ISIC. There, DiFF reports diagnostic AUC gains of up to 2.51% and 2.04% over consensus baselines (Wu et al.). No reported absolute AUC values were found on Lacuna for those benchmarks, so we will rerun DiFF ourselves under matched budgets.