Many clinical prediction models follow the same recipe. A doctor outlines a tumor by hand, software computes a few thousand texture and shape numbers from inside the outline, and a linear support vector machine (a classifier that places a flat decision boundary as far as possible from the nearest training cases) predicts a label such as malignancy or invasiveness. The weak point is the outline. Two radiologists tracing the same nodule shift the boundary by a few millimetres, and the texture numbers move with it. Current pipelines deal with this bluntly. They measure how consistent each feature is across readers and delete any feature that falls below a cutoff. Under manual tracing, that cutoff can remove about half the feature pool, and it tends to remove the boundary-texture features that may carry signal about invasion.
We propose a gentler fix. First, learn from multi-reader data which directions in feature space contour disagreement pushes cases along. Summarize this as a covariance matrix of rater-induced feature shifts. Then train a linear SVM that must keep every case correctly classified even after the worst plausible shift of that shape. The result is a classifier whose weights are shrunk along the directions readers disagree on, instead of losing whole features. Because the shift covariance is estimated once, on a public multi-reader dataset, it can be carried to the common single-contour cohort. The claim to test is that this beats feature deletion and simple augmentation on the thing that matters clinically: whether a patient's prediction changes when a different doctor draws the outline.
Core Claim
Suppose each case's feature vector $x_i$ may be displaced by any shift $\Sigma^{1/2}u$ with $\|u\|\le\rho$. Here $\Sigma$ is the covariance of feature differences between readers' contours of the same lesion, and $\rho$ is a radius. Taking the worst case of the hinge loss over this ellipsoid gives
$$\min_{w,b}\ \tfrac12\|w\|^2 + C\sum_i \max\!\big(0,\ 1 - y_i(w^\top x_i+b) + \rho\sqrt{w^\top \Sigma w}\big),$$
where $w,b$ are the boundary parameters, $y_i\in\{-1,+1\}$ the label, and $C$ the usual slack weight. The extra term $\rho\sqrt{w^\top\Sigma w}$ penalizes weight placed on directions that contour disagreement moves. The problem remains convex and solves by subgradient descent or as a second-order cone program.
Two special cases show where this sits. With $\Sigma = I$ it reduces to an isotropic robust SVM, which is essentially what Wasserstein Distributionally Robust Multiclass Support Vector Machine optimizes. That paper builds a worst-case ball around training points with a generic distance plus a label-flip cost, and tests it only on tabular UCI and synthetic data. With $\Sigma$ diagonal and an infinite penalty above a variance threshold, it reduces to intraclass-correlation filtering, the standard practice. Our contribution is the middle ground: a full, low-rank-plus-diagonal $\Sigma$ estimated from real reader contours and transferred across cohorts. The central empirical question is whether the off-diagonal structure matters. If a diagonal $\Sigma$ does as well, the idea reduces to per-feature rescaling, and we will report that.
Motivation from Prior Work
Three lines of evidence motivate the gap. Robust Radiomics Feature Quantification Using Semiautomatic Volumetric Segmentation compared five physicians' manual lung-tumor contours against semiautomatic ones. Only 52% of 56 features reached ICC ≥ 0.8 under manual tracing, compared with 70% under semiautomatic segmentation. The study stopped at reproducibility statistics and never asked what the instability does to a classifier.
Two SVM radiomics studies show the downstream practice. Radiomics model building from multiparametric MRI to predict Ki-67 expression in PCNSL filtered features by ICC before training an SVM on 83 patients. Prediction of high infiltration levels in pituitary adenoma deliberately extracted features from a dilated boundary region, then trained a linear SVM without checking whether its 19 LASSO-selected features survive a different neurosurgeon's contour. Together these define our target regime: few hundred cases, thousands of features, one contour per case.
On the segmentation side, Probabilistic Modeling of Inter- and Intra-observer Variability (Pionono), Variational Inference for Quantifying Inter-observer Variability and Modeling Multimodal Aleatoric Uncertainty with Mixture of Stochastic Experts all model reader disagreement as a distribution over masks. They evaluate only on mask overlap and never propagate that distribution into a classifier. We use their premise, that disagreement is structured rather than blur, but operate on features rather than pixels.
Diagnostic subjectivity enters mainly as a constraint on evaluation. Re-thinking and Re-labeling LIDC-IDRI showed that the visual malignancy scores in LIDC-IDRI are biased toward suspicion. Against biopsy-anchored labels, the original consensus had sensitivity 0.97 and specificity 0.23, and more than half of nodules scored 4 were re-labeled benign. A classifier trained and scored only on consensus labels therefore partly learns reader habit, so we cannot treat consensus accuracy as the whole story.
The Latent Doctor Model models per-reader label style with a conditional generative network. It is the natural label-side comparison, but it targets large patch datasets rather than cohorts of a few hundred.
Research Program
The primary testbed is LIDC-IDRI, the one public dataset in the file where up to four radiologists both outlined each nodule and scored its malignancy. It therefore lets us measure contour disagreement and label disagreement on the same cases.
For nodules with at least three contours, we extract a standard feature set of intensity, shape and texture features from each reader's mask. We then form the within-nodule difference vectors and fit $\Sigma$ as a low-rank-plus-diagonal covariance, so that it stays estimable with $d$ in the thousands. The fit uses a disjoint set of nodules from those used to train and test the classifier.
The classifier is trained under a deliberately realistic handicap. It sees one randomly chosen reader's contour per training nodule, which mirrors single-contour cohorts like the PCNSL and pituitary studies. We repeat training at subsampled sizes of roughly 50, 100 and 200 nodules to match their sample sizes. $\rho$ and $C$ are tuned by nested cross-validation.
As a secondary analysis, we compute each test case's worst-case margin, meaning its margin minus $\rho\sqrt{w^\top\Sigma w}$, as a contour-fragility score. We then ask whether contour-fragile nodules are the same ones where radiologists disagree on malignancy score. If they are not, contour noise and diagnostic subjectivity are separable sources of error that need separate handling.
The label side receives one light-touch treatment that keeps the model linear. Each nodule's slack weight is scaled by reader agreement on its malignancy score, so contested cases pull less on the boundary. This is reported as an ablation, not a headline.
A transfer study tests how general $\Sigma$ is. We estimate shift covariances separately on multi-reader QUBIQ tasks (QUBIQ challenge), specifically kidney CT, pancreas CT and pancreatic lesion CT, and measure how well their leading directions align with the LIDC-IDRI $\Sigma$. If the dominant shift directions are shared across CT organs, and track features known to be boundary-sensitive, a single estimated $\Sigma$ becomes a reusable asset for CT radiomics. If they are not shared, the method requires a small multi-reader calibration set per application, which we would state as its cost.
Baselines
The naive reference is a standard soft-margin linear SVM trained on all features from the single available contour, with no stability handling. It sets the floor for both accuracy and cross-reader prediction flips.
The strongest existing method in practice is ICC filtering followed by a linear SVM, the pipeline used in the PCNSL study. Within the file it is the only reported SVM result in this regime, with a test AUC of 0.869 on a private cohort. Because that cohort is not available, the number is context, not a comparison target. We run this baseline on LIDC-IDRI with ICC computed on the same held-out multi-reader nodules used to fit $\Sigma$, sweeping the threshold (0.75 and 0.8, the cutoffs the literature uses) within the same nested cross-validation budget.
The second strong competitor is the isotropic robust SVM, the $\Sigma=I$ case matching the geometry of WDR-MSVM. It is tuned over $\rho$ identically. No reported result was found on Lacuna for a linear SVM radiomics classifier on LIDC-IDRI, so all baseline numbers here will be produced under our protocol.
The key-ingredient ablation is a diagonal $\Sigma$ built from per-feature inter-reader variance. It isolates whether the correlated structure of contour shifts carries information beyond per-feature noise levels.
Beating the baselines means a lower cross-reader flip rate at equal or higher mean AUC than ICC filtering and the diagonal ablation, consistently across training sizes and with confidence intervals from repeated splits that do not overlap zero difference.
Why not just…?
Why not just filter by ICC and move on? It is cheap and accepted practice. In our formulation it is the degenerate diagonal, infinite-penalty case, and it discarded 48% of features under manual tracing in the 3D-Slicer study. Whether that loss hurts prediction is untested. Our head-to-head at matched tuning budget settles it. If ICC filtering matches us on both AUC and flip rate, the proposal fails on its main claim.
Why not just augment training with every reader's contour, or with eroded and dilated masks? This is the most serious threat. Training on all reader copies optimizes the average hinge over readers, whereas our objective controls the worst shift. Averaging over only three or four copies spans a tiny subspace of a thousand-dimensional feature space. The more important limitation is that reader-copy augmentation is impossible in the single-contour cohorts that motivate the work. Morphological augmentation is possible there, but it assumes readers disagree by uniform erosion or dilation.
We therefore include three augmentation variants as full baselines under identical protocol. The first uses all reader copies, with access to extra contours the proposed method does not see at training time. The second uses synthetic erosion and dilation of the single contour. The third takes the max over copies. If synthetic morphological augmentation matches $\Sigma$-shaped training, we will conclude that the empirical rater geometry is unnecessary, and redirect the work toward characterizing which perturbation family best imitates readers.
Why not just use a deep multi-rater model? Models like the Latent Doctor Model and Pionono target large patch or pixel datasets and model label or mask distributions. They do not target feature-level robustness of a small-sample classifier. At $n\le200$ with thousands of features, a linear model is the realistic comparator. We therefore do not claim to beat these models on their own benchmarks, and we note that they solve a different problem.
Decisive Checks
The idea stands or falls on four measurements taken on held-out LIDC-IDRI nodules scored under every available reader's contour. The first is mean AUC against consensus labels, excluding score-3 nodules as is conventional. The second is the spread of AUC across readers. The third is the fraction of nodules whose predicted class flips when the contour is swapped. The fourth, where the biopsy-anchored relabeling of Re-thinking and Re-labeling LIDC-IDRI can be obtained, is performance on those labels, so that robustness to contours is not bought by fitting reader bias more tightly.
The most important single comparison is full versus diagonal $\Sigma$. The claim that contour disagreement has exploitable geometry beyond per-feature variance is true only if the full $\Sigma$ wins. A second decisive check is whether $\Sigma$ estimated on other nodules, or on QUBIQ CT organs, still reduces flip rate. That determines whether this is a reusable regularizer or a per-dataset calibration step.
References
- Wasserstein Distributionally Robust Multiclass Support Vector Machine
- Robust Radiomics Feature Quantification Using Semiautomatic Volumetric Segmentation
- Radiomics model building from multiparametric MRI to predict Ki-67 expression in patients with primary central nervous system lymphomas: a multicenter study
- Prediction of high infiltration levels in pituitary adenoma using MRI-based radiomics and machine learning
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures
- Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
- The Latent Doctor Model for Modeling Inter-Observer Variability
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge