Radiomic models predict a diagnosis from hundreds or thousands of numbers computed inside a hand-drawn outline of a tumor or organ. A single expert usually draws that outline, and a selection procedure such as LASSO or stepwise regression then keeps a handful of predictive numbers. A different expert would draw a slightly different outline. The numbers would change, and the chosen model might give a different answer for the same patient. We propose a feature-selection procedure that greedily adds and removes features, scoring each candidate subset by how well a model fitted on one expert's outlines predicts outcomes for new patients outlined by a different expert. The procedure keeps a feature only if it improves prediction that survives a change of annotator.
The underlying point is that clinicians care whether the model's decisions are reproducible across annotators, not only whether each feature is. The standard remedy, discarding features whose values disagree across raters, looks at one feature at a time. It can keep features whose small disagreements add up to a large shift in the prediction, and it can discard features whose disagreements do not change patient rankings. Scoring subsets directly on cross-annotator transfer targets the quantity that matters. The result is still a short list of interpretable features, which end-to-end deep multi-rater methods do not provide.
Core Claim
Variable selection on features from multi-rater segmentations should treat annotators as a distribution shift and choose subsets by their out-of-rater predictive performance. Annotator variability should not be handled by filtering features beforehand. We claim that a forward–backward (bidirectional) stepwise search with this score yields models that have, at matched in-rater accuracy, higher accuracy on held-out raters and fewer prediction flips when the contour changes. We also claim that the advantage over per-feature reproducibility filtering grows with inter-rater disagreement.
Relation to Prior Work
Three groups of work frame the gap.
The radiomics studies motivate it. Chen et al. selected 5 of 546 CT features with LASSO from single-reader thyroid contours. Zhang et al. filtered 4,120 MRI features down to 19 before an SVM. Both report that contour subjectivity alters texture features, yet neither tested whether the selected set survives a different reader. Radiomics and Its Feature Selection: A Review notes that segmentation-induced instability is rarely built into wrapper or stepwise search. That observation is exactly the slot we fill.
The stepwise literature supplies the method and its known weaknesses. Bidirectional search with information criteria, as used by Hoshikuma et al. and Mori et al., assumes predictors are measured without error. Stepwise selection is also notoriously unstable under small data perturbations. We keep the search procedure and replace the score.
The multi-rater modeling literature supplies both a competitor and a tool. Wu et al.'s DiFF learns a diagnosis-weighted fusion of rater masks inside a deep network and reports diagnostic AUC gains. It explicitly leaves open how this relates to interpretable biomarkers. We share its "the downstream task defines the right segmentation" stance but return an explicit feature list. Generative rater models, namely Pionono, the Latent Doctor Model and the variational model of Liu et al., produce plausible rater-specific masks but stop at segmentation. We use them only in a secondary arm, to synthesize extra "raters" when few real ones exist.
Zhang et al.'s LIDC-IDRI re-labeling constrains the design. They show that consensus labels can carry systematic bias: their baseline model had sensitivity 0.97 and specificity 0.23. Rater-swap selection therefore addresses contour variability only. It does not correct biased outcome labels, and we evaluate it separately against biopsy-anchored labels where possible.
Research Program
The setting is as follows. Each patient $i$ has contours from raters $r = 1,\dots,R$ and an outcome $y_i$. Pyradiomics-style features $x_i^{(r)}$ are extracted from each contour. For a candidate feature subset $S$, we fit a regularized logistic model on training patients using rater $a$'s features. We then evaluate it on held-out patients using rater $b$'s features ($b \neq a$). The selection score is a lower quantile of these cross-rater AUCs:
$$ Q_\tau(S) = \text{quantile}_\tau \big\{ \mathrm{AUC}_{a \to b}(S) : a \neq b,\ \text{over CV folds} \big\}, $$
where $\tau$ (for example 0.25) sets how pessimistic the score is, and $\mathrm{AUC}_{a\to b}$ is the area under the ROC curve when training on rater $a$ and testing on rater $b$.
The search is bidirectional. At each step it adds the feature that most raises $Q_\tau$. It then removes any earlier feature whose deletion does not lower $Q_\tau$. It stops when the best gain is smaller than one bootstrap standard error of $Q_\tau$. This one-standard-error rule replaces BIC, since BIC has no notion of rater shift. Patients and raters are both resampled in the bootstrap so that the stopping rule reflects both sources of noise.
We will build and release this procedure. We will also analyze why it chooses differently from reproducibility filters by decomposing each subset's cross-rater prediction shift into per-feature terms and covariance terms. The covariance terms are the part a univariate filter cannot see. In the synthetic-rater arm, Pionono or the Latent Doctor Model is trained on the multi-rater data, and samples from its latent rater space serve as additional $b$'s. We test whether synthetic raters predict real held-out-rater performance. If they do, the method extends to the common single-reader cohorts like those of Chen et al. and Zhang et al.
Baselines
The naive reference point follows Chen et al.'s pipeline: build a consensus mask by majority vote, extract features once, select with LASSO, and fit logistic regression. This is what most published radiomic signatures do. Any proposal must beat it on held-out raters, or the problem is not real.
The strongest simple competitor is reproducibility filtering followed by LASSO. Features are kept only if their intraclass correlation across raters exceeds a threshold, and LASSO is then run on consensus features. This is widespread practice, though no paper in the file reports it on these benchmarks, so we have no reported number. We will tune its threshold over the same grid and CV budget as our $\tau$.
A second strong competitor is rater-pooled LASSO, which stacks every rater's feature vector as an augmented training example. It implicitly asks for rater-invariant coefficients. Since it is the most plausible equal-performer, we tune it identically. We also include stability selection over joint patient-and-rater bootstraps.
The deep reference is DiFF. On REFUGE-2, TNMIX and ISIC, Wu et al. report diagnostic AUC improvements of up to 2.73%, 2.51% and 2.04% respectively over baseline multi-rater methods. No absolute AUC is reported in the file, so we will rerun DiFF ourselves on identical splits. We do not expect a sparse linear model to beat DiFF's in-distribution AUC. Matching it within its reported margin on held-out raters while using under twenty named features would count as success.
The key ablation runs our exact search with an in-rater score, training and testing on the same rater's features. This isolates the cross-rater scoring as the active ingredient. All methods share folds, feature extraction, final classifier family, and a fixed number of hyperparameter evaluations.
"Beating" a baseline means two things. First, a statistically significant gain in held-out-rater AUC, measured by a paired bootstrap over patients. Second, a lower flip rate, defined as the fraction of patients whose thresholded prediction changes when their contour is swapped between raters. Both must hold at an in-rater AUC no worse than 0.01 below the baseline.
Why not just…?
Why not just filter features by inter-rater reproducibility and then run LASSO? This is the real threat. If univariate reproducibility were sufficient, our method would add only compute. Our argument is that prediction stability depends on how rater errors co-vary across the selected features. When one rater draws systematically larger outlines, shape and texture features shift together, and their combined effect on the linear score can be large even when each feature looks reproducible. The covariance decomposition in the research program measures this directly. If the filter matches our held-out-rater AUC and flip rate on every dataset, we will report that the simpler practice suffices.
Why not just pool all raters' contours as augmented training data? Pooling does push coefficients toward rater invariance, and it may perform close to our method. It optimizes average fit, however, not worst-case transfer, and it never tests on unseen raters during selection. Our prediction is that pooling helps most when raters differ randomly and our method helps most when one or two raters are systematically biased. The per-dataset correlation between our advantage and measured rater bias settles this.
Why not just use DiFF or a deep ensemble end-to-end? These may well give higher in-distribution AUC. The QUBIQ summary shows that ensembles are strong but costly on 3D data, and none of these methods yields an auditable feature list. We report DiFF honestly as a ceiling. Our claim concerns reproducibility per feature retained, not raw accuracy.
Why not just segment automatically and remove the observer? An automated segmenter trained on consensus masks inherits one perspective, as the LIDC-IDRI bias result illustrates for labels. We include the automated mask as an extra "rater". Our method should select features that also transfer to it, which we check directly.
Proposed Experiments
The primary datasets are REFUGE-2, TNMIX and ISIC, which provide multi-rater masks and diagnostic labels on DiFF's splits, and LIDC-IDRI, which has at least three radiologist contours per nodule. QUBIQ (seven raters on brain growth) and Gleason 2019 (six pathologists) are used only to train and calibrate the synthetic-rater generators. Pionono's reported Cohen's kappa of 0.758 on Gleason 2019 serves as the bar a reimplementation must reach before its samples are trusted.
The decisive check is leave-one-rater-out evaluation. One rater is excluded from all selection and fitting. Models are evaluated only on that rater's contours for held-out patients, which removes any leakage from scoring and testing on the same raters. A second check addresses label bias on LIDC-IDRI. Models are scored against both consensus malignancy and the biopsy-propagated labels of Zhang et al.. This confirms that contour robustness does not trade off against label correctness.
The third check tests the dose-response prediction. Across datasets and across artificially perturbed rater sets with controlled mean Dice between raters, the gain of rater-swap selection over the reproducibility filter should increase as agreement falls. If no such trend appears, the mechanism we propose is wrong even if the average gain is positive.
References
- Computed Tomography Radiomic Nomogram for Preoperative Prediction of Extrathyroidal Extension in Papillary Thyroid Carcinoma
- Prediction of high infiltration levels in pituitary adenoma using MRI-based radiomics and machine learning
- Radiomics and Its Feature Selection: A Review
- Statistical Analysis of the Factors that Affect Postoperative Length of Hospital Stay after Unruptured Intracranial Aneurysm Treatment in Japan: A 20-year Nationwide Multicenter Study
- Multifaceted evaluation of central post–stroke pain
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge