Radiomics computes thousands of texture statistics inside a hand-drawn tumor outline and uses them to predict things a radiologist cannot see, such as malignancy or mutation status. The weak point is the outline itself. Two radiologists never draw the same boundary, and the texture statistics that depend on that boundary can swing enough to change a prediction. The field's standard remedy is a reproducibility study, in which several physicians contour the same patients and only features with high agreement across physicians are kept. This study is expensive, and it has to be repeated for every new organ, scanner and cohort. It is the annotation bottleneck in its most acute form.
We propose to replace the human reproducibility panel with a learned one. We would train a segmentation model to reproduce the spread of outlines that real radiologists produce, not a single consensus mask. On any new scan, the model then generates a "virtual panel" of plausible outlines, conditioned only on a cheap input such as a single diameter line. Running the radiomics extractor over that panel tells us, per patient and per feature, how much each feature would move under human disagreement. That estimate lets us keep or average features without collecting new multi-reader contours. We would judge the resulting features against biopsy-proven malignancy rather than against subjective reader scores, because those scores are known to be biased.
Core Claim
Inter-observer contour variability is structured. It concentrates at spiculated, infiltrative edges and is small along smooth borders. A model that has learned this structure from an existing multi-reader dataset can do two things. First, it can predict the feature-level reproducibility that a real multi-physician study would measure. Second, it can supply mask-averaged features that are more stable and at least as predictive of pathology as features from any single mask. Both claims can be falsified. The first fails if the virtual panel's per-feature reproducibility does not track real-reader reproducibility better than simple geometric perturbation of one mask. The second fails if mask averaging does not beat a consensus mask on biopsy-labeled nodules.
Concretely, let $x$ be a scan, $a$ the cheap annotation (for example a long-axis line), and $p_\theta(m \mid x, a)$ the learned distribution over binary masks $m$. For radiomic feature $f_k$, the virtual panel yields both a mask-averaged feature and a virtual intraclass correlation:
$$\bar f_k(x) = \mathbb{E}_{m \sim p_\theta(m\mid x,a)}\big[f_k(x,m)\big], \qquad \widehat{\mathrm{ICC}}_k = \frac{\sigma^2_{\text{between},k}}{\sigma^2_{\text{between},k} + \mathbb{E}_x\big[\mathrm{Var}_{m\sim p_\theta}\, f_k(x,m)\big]}.$$
Here $\sigma^2_{\text{between},k}$ is the variance of $\bar f_k$ across patients, and the second term is the average within-patient spread caused by sampled outlines. This is the same ratio a real reproducibility study computes, with sampled masks standing in for physicians.
Relation to Prior Work
Three papers motivate the gap. The first is Robust Radiomics Feature Quantification Using Semiautomatic Volumetric Segmentation, which had five physicians contour 20 lung cancer CTs. It reports a mean feature ICC of 0.77 ± 0.17 for manual contours and 0.85 ± 0.15 for GrowCut region growing, with 52% versus 70% of features reaching ICC ≥ 0.8. It also documents that region growing smooths away spiculations, the edges that carry invasive biology. The second and third are clinical radiomics studies of thyroid extrathyroidal extension (CT Radiomic Nomogram for Extrathyroidal Extension) and pituitary invasion (Prediction of high infiltration levels in pituitary adenoma). Both name manual ROI variability as a primary limitation. All three treat reproducibility as something measured once by humans. We treat it as something a model predicts for new cases.
The method comes from multi-rater segmentation. Pionono assigns each rater a Gaussian in a latent space and decodes coherent alternative masks. Disentangling Human Error from Ground Truth jointly estimates a true mask and per-annotator confusion matrices. MRNet produces pixel-wise uncertainty maps from multi-rater agreement. We reuse Pionono-style latent sampling as the mask generator. These works stop at Dice and calibration of the mask. None of them sends the sampled masks through a texture extractor, and the research file confirms this as an open gap.
Two papers constrain the design. DiFF fuses multi-rater masks with weights chosen to maximize a diagnostic classifier trained on clinical labels. Re-thinking and Re-labeling LIDC-IDRI shows that clinical malignancy scores carry strong risk-averse bias: a model trained on raw LIDC-IDRI scores reached 0.97 sensitivity but 0.23 specificity against biopsy truth, and re-labeling through 180 biopsy-proven anchors moved this to 0.67/0.66. Together these warn that tuning masks to diagnostic labels can bake reader bias into the boundary. We therefore never optimize masks for a diagnostic label. The generator is trained only to match contour disagreement, and diagnosis is evaluated only against pathology.
On the annotation side, Weakly Supervised Lesion Co-Segmentation on CT Scans turns routine RECIST diameter lines into masks with 90.3% mean Dice on DeepLesion. It does not ask whether those masks give stable textures. Radiomics-Guided Global-Local Transformer and invasion-weakly-supervised PitNET radiomics take a different route: the first uses attention-derived boxes and the second a single frozen nnU-Net mask. Neither represents boundary uncertainty. We borrow the RECIST-line conditioning idea and add the uncertainty those pipelines discard.
Research Program
The work has three parts.
First, we train the virtual panel on LIDC-IDRI. Up to four thoracic radiologists outlined each nodule there. The generator is a 3D U-Net with a Pionono-style latent variable. It is conditioned on the CT crop and on a synthetic long-axis line derived from the reader masks, so that at deployment it needs only a RECIST-like measurement. LIDC reader identities are not reliably linked across scans. We therefore treat each reader slot as an anonymous annotation and fit a shared latent distribution over annotations rather than per-person styles.
Second, we build the radiomics layer. A fixed Pyradiomics configuration (shape, first-order, co-occurrence, run-length and wavelet families) is run on 20 to 50 sampled masks per nodule. This yields the mask-averaged features $\bar f_k$ and the virtual ICC per feature. We also record per-patient feature spread, which serves as a confidence signal.
Third, we analyze where the virtual panel places its variance. We test whether within-patient feature spread concentrates in irregular-margin nodules, defined objectively by low mask compactness. We compare against isotropic erosion and dilation, which by construction cannot distinguish a spicule from a smooth edge. This is the mechanism behind the core claim. If learned variance is not anisotropic, the proposal reduces to geometric perturbation.
Baselines
The naive reference is the current default: features computed from one reader's mask (or the generator's single most likely mask), filtered by LASSO, with no reproducibility screening. It shows what happens when variability is ignored.
The strongest existing practice is consensus-mask radiomics: build a STAPLE or majority-vote mask from the available readers, extract features, and screen them by the real multi-reader ICC computed on the training split. This is what a careful radiomics group would do. It uses the expensive human panel that we aim to replace, so matching it without that panel at test time already counts as a win. On reproducibility, the published reference points are the manual ICC of 0.77 ± 0.17 and semiautomatic ICC of 0.85 ± 0.15 from Velazquez et al., measured on a different cohort, so they are context rather than a head-to-head target. On diagnosis, the strongest reported number on this data is the re-labeled deep model's 0.67 sensitivity and 0.66 specificity from Re-thinking and Re-labeling LIDC-IDRI. We know of no reported biopsy-validated radiomics result on SCH-LND, so that comparison will be run here.
The direct competitor for the key ingredient is single-mask geometric perturbation: erode, dilate and randomly deform one mask, then compute ICC from the perturbed copies. It needs no multi-rater training data. A deep ensemble of deterministic U-Nets serves as a second mask sampler, because QUBIQ found such ensembles often match probabilistic architectures, with top entries at 0.80–0.85 average Dice. We also run a mask-free end-to-end 3D CNN on nodule crops, trained on the same labels.
Two ablations isolate the ingredients. The first removes the learned latent, so samples come only from dropout noise. The second uses virtual ICC for feature screening but drops mask averaging, which separates the selection effect from the averaging effect.
All methods share one Pyradiomics configuration, one feature-selection and logistic-regression pipeline with the same hyperparameter grid, identical patient splits, and an equal number of sampled masks for every sampler. Beating the baselines means two things. On reproducibility, the virtual ICC must correlate with held-out real-reader ICC significantly better than geometric perturbation and ensembles do. On diagnosis, it must give a higher AUC on biopsy-proven nodules than consensus-mask radiomics, with non-overlapping bootstrap intervals or a significant DeLong test.
Why not just…?
Why not just erode and dilate one mask to estimate feature robustness? This is the most serious threat, because it is cheap and needs no multi-rater data. Its weakness is that it moves every boundary voxel equally. Real readers mostly disagree at spicules, which is exactly where Velazquez et al. found automated methods failing. The decisive test is the held-out comparison of predicted against real-reader per-feature ICC. If geometric perturbation tracks real ICC as well as the learned panel, we will report that and recommend perturbation, and the contribution narrows to the diagnostic evaluation.
Why not just use a deep ensemble instead of a rater-variability model? QUBIQ reports that ensembles often beat probabilistic models on mask agreement, so they are a real contender as samplers. However, an ensemble's spread reflects model uncertainty, which shrinks with more data, whereas reader disagreement does not. The ensemble is run as a drop-in sampler in every experiment. If it matches the learned panel on ICC prediction, the conclusion is that any reasonable sampler suffices, which is still a useful finding.
Why not just take a consensus mask with STAPLE? Consensus collapses disagreement into one outline. Disentangling Human Error from Ground Truth reports that STAPLE trails learned methods by 14.4% in confusion-matrix error on BraTS, and consensus also needs several readers on every new case. Consensus-mask radiomics is our main diagnostic baseline, so the experiments settle directly whether averaging over sampled masks adds anything.
Why not skip radiomics and train a 3D CNN end to end? With only 180 biopsy-proven anchors in SCH-LND, deep models depend on large label sets, and those labels are biased, as the 0.23 specificity in Re-thinking and Re-labeling LIDC-IDRI shows. The end-to-end CNN is included under the same splits. If it wins on pathology AUC, mask-based radiomics is not worth rescuing for this task, and we will say so.
Proposed Experiments
The first decisive check is reproducibility prediction on LIDC-IDRI. The generator is trained on one patient split. On held-out nodules that have at least three reader contours, we compute the real per-feature ICC from the readers and the virtual ICC from samples, then report three things across the feature set: Spearman correlation between the two, agreement on the ICC ≥ 0.8 high-reproducibility set measured with Cohen's kappa, and these results stratified by mask compactness to test the spiculation hypothesis.
The second check is diagnosis against pathology. Models are trained on LIDC-IDRI with the re-labeled labels of Zhang et al. and evaluated on SCH-LND biopsy-confirmed nodules. Masks on SCH-LND come only from the generator, conditioned on a single annotated long-axis line. Because those re-labels were derived from SCH-LND anchors, evaluation uses an anchor/test partition of SCH-LND, with re-labeling redone using only the anchor half to avoid leakage. We report AUC, sensitivity and specificity, plus calibration of predicted probability against per-patient feature spread. We also run a second arm trained on raw subjective scores, to measure how much reader bias each pipeline inherits.
The third check tests the cheap-annotation regime. On DeepLesion lung lesions, we condition the generator on the real RECIST lines and report Dice against the manual reference masks used by Agarwal et al., whose reported number is 90.3% mean Dice. We also report virtual-ICC distributions to show whether line-conditioned masks yield usable textures. No outcome labels exist there, so this check speaks only to whether the method transfers to routine clinical measurements, not to diagnosis.
References
- Robust Radiomics Feature Quantification Using Semiautomatic Volumetric Segmentation
- Computed Tomography Radiomic Nomogram for Preoperative Prediction of Extrathyroidal Extension in Papillary Thyroid Carcinoma
- Prediction of high infiltration levels in pituitary adenoma using MRI-based radiomics and machine learning
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Disentangling Human Error from Ground Truth in Segmentation of Medical Images
- Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
- Weakly Supervised Lesion Co-Segmentation on Ct Scans
- Radiomics-Guided Global-Local transformer for weakly supervised pathology localization in chest X-Rays
- An MRI radiomics approach using invasion-based weak supervision for identifying and evaluating aggressive PitNETs
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge