Variance-Penalized Radiomics: Keeping Fragile but Predictive Features Without Unreliable Predictions

Radiomic models turn a medical scan into hundreds or thousands of numbers that describe a tumor, such as its intensity histogram, its shape, and its texture, and then predict an outcome like survival from those numbers. Many of these numbers change when the image is nudged slightly: a small rotation, a little added sensor noise, or a contour drawn a few pixels wider. The usual remedy is to measure how stable each feature is and discard the unstable ones. That remedy carries a known cost. On a head-and-neck PET/CT survival task, keeping only features whose test–retest agreement exceeded 0.75 raised the agreement of the model's own predictions from 0.596 to 0.825, but the test C-index fell from 0.769 to 0.641 (Building Reliable Radiomic Models Using Image Perturbation). The most predictive features were among the most fragile.

This proposal starts from the logic of inverse-variance weighting, the rule that when you combine noisy estimates you should trust each one in proportion to one over its variance. Applied to radiomics, that logic suggests shrinking each feature in proportion to how much it moves under perturbation, instead of keeping or deleting it outright. The key observation is that a linear risk score $r_i = \beta^\top x_i$ for patient $i$ moves under image perturbation by exactly $\beta^\top \Sigma_i \beta$, where $\beta$ is the coefficient vector and $\Sigma_i$ is the covariance of that patient's feature vector across perturbed copies of the scan. Penalizing this quantity during training therefore controls the reliability metric directly, whereas an ICC filter only approximates it. We claim that a penalty built from measured perturbation covariance traces a better trade-off between reliability and accuracy than hard filtering. The test is whether, at test reliability near 0.8, this penalty keeps C-index well above 0.641.

Core Claim

Let $\bar\Sigma = \frac{1}{n}\sum_i \Sigma_i$ be the perturbation covariance averaged over the $n$ training patients, estimated from each patient's perturbed scans. We fit a Cox model by maximizing

$$\ell(\beta) - \lambda\, \beta^\top \bar\Sigma \beta,$$

where $\ell$ is the Cox partial log-likelihood and $\lambda \ge 0$ sets how much within-patient wobble in the risk score we tolerate. When $\bar\Sigma$ is diagonal, the penalty shrinks each coefficient in proportion to its feature's perturbation variance. That is inverse-variance weighting carried over from pooling estimates to fitting coefficients. When $\bar\Sigma$ is the full matrix, the model can keep combinations of fragile features whose perturbation errors cancel, for example two texture measures that shift in opposite directions when the contour grows. A per-feature ICC cannot see this case, and we expect it to explain part of the gap. Sweeping $\lambda$ produces a curve of test C-index against test reliability. The claim is that this curve lies above the single point the ICC filter reaches, and above the curve traced by ordinary ridge shrinkage, which penalizes all features equally regardless of how fragile they are.

There is a second, patient-level ingredient. Each patient's own $\Sigma_i$ gives a measured variance for that patient's predicted risk, $v_i = \beta^\top \Sigma_i \beta$. Small or irregular tumors should have much larger $v_i$ than large, compact ones. This measured quantity is exactly what Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression assumed was known in advance and named as unresolved for real data. That paper showed that weighting each sample's loss by inverse variance only trains stably if the weights are normalized within each mini-batch and floored by a small constant. We reuse that recipe to down-weight high-$v_i$ patients' contributions to the partial likelihood. We also report $v_i$ at test time so that predictions too unstable to act on can be flagged.

Relation to Prior Work

The perturbation study above motivates this proposal and supplies its protocol. It produced 60 perturbations per scan, combining geometric shifts, Gaussian noise, and contour deformation, and it measured the trade-off. It handled instability with a binary cutoff. We keep its perturbations and replace the cutoff with a continuous penalty. Radiomics Analysis Using Stability Selection Supervised Principal Component Analysis for Right-censored Survival Data also selects features for survival radiomics, but its notion of "stable" is stability across patient resamples, which is a different variance from image perturbation. It is therefore a competing selection method rather than a substitute for ours.

Two papers bring inverse-variance weighting into imaging, and neither applies it to radiomic features. FIVA weights neural network parameter updates by their variance during training when combining models across hospitals for CT segmentation. Batch Inverse-Variance Weighting weights regression losses on face photographs with synthetic label noise. Our setting differs from both: the noise sits in the input features rather than in labels or parameters, and its variance is measured directly from perturbations rather than assumed. Both papers constrain our design. BIV's finding that naive inverse weights fail to converge is why the patient-level weights are batch-normalized and floored. FIVA's report that variance estimates become noisy when data are scarce is a warning about estimating $\Sigma_i$ from 60 perturbations of a few hundred features. For that reason we use a shrinkage estimator for each $\Sigma_i$ and treat the diagonal-versus-full comparison as an experiment rather than an assumption.

AI-driven preclinical disease risk assessment using imaging in UK biobank found that tabular radiomics with tree models and MLPs beat a 3D ResNet18 on whole-body MRI. That supports studying hand-crafted features rather than abandoning them.

Research Program

We will reimplement the perturbation pipeline on the Head-Neck-PET-CT collection: 298 patients, the same 208/90 train–test split as the perturbation study, 60 perturbations per scan, and features extracted from the original images, their wavelet filters, and their Laplacian-of-Gaussian filters. For every patient we store the perturbed feature matrix. From it we estimate a shrunk $\Sigma_i$ and the pooled $\bar\Sigma$.

We then fit four estimators along matched regularization paths. The first is the full-covariance penalty. The second is the diagonal penalty, which is the direct inverse-variance analogue. The third adds BIV-style patient weights on top of the full-covariance penalty. The fourth, a control, keeps the penalty's overall size but randomly permutes the measured variances across features. For every value of $\lambda$ we record test C-index and test reliability. Reliability is the ICC of the risk score across the 60 perturbed test copies, which is the same metric the perturbation study used.

As a secondary, real-world check we use LIDC-IDRI. There, independent radiologists' contours of the same nodule supply natural contour perturbations, so $\Sigma_i$ comes from observed reader disagreement rather than synthetic deformation. On this dataset we predict malignancy from tabular features.

Three analyses support the main comparison. First, we inspect which fragile features survive under the full-covariance penalty but are cut by the ICC filter, and check whether they are cancelling pairs. Second, we test whether patients with high $v_i$ are ordered less accurately by the model, using C-index computed within bins of $v_i$. Third, we test whether withholding the most unstable 20% of test predictions raises C-index on the rest more than withholding a random 20%.

Baselines

The naive reference is the unfiltered baseline model of the perturbation study. It reports a test C-index of 0.769 with a test reliability ICC of 0.596. The closest existing method is its ICC > 0.75 filter, which reports a test C-index of 0.641 with a test reliability ICC of 0.825 (Building Reliable Radiomic Models Using Image Perturbation). Our numbers will come from a reimplementation and may not match these exactly. We will therefore rerun both baselines in our own pipeline, using the same features, split, and perturbations, and compare against the rerun numbers. The published figures serve only as a sanity anchor.

We also run the ICC filter at several thresholds, not just 0.75, so that it traces a curve too. The comparison is then curve against curve, which is fair to the filter.

Plain ridge and elastic-net Cox are the strongest generic competitors, because shrinkage of any kind dampens prediction wobble. They receive the same $\lambda$ grid and the same nested cross-validation budget as our method. Stability-selection supervised PCA (Radiomics Analysis Using Stability Selection Supervised Principal Component Analysis for Right-censored Survival Data) is run with its resampling settings tuned on the training set. For context, that paper reports Uno's C-index of 0.628 on its own NSCLC cohort and 0.705 on its own head-and-neck cohort. Those are different datasets, so the numbers are not directly comparable to ours.

Training a Cox model on all perturbed copies of each patient is the most dangerous baseline, and it is discussed below.

The permuted-variance control isolates the key ingredient. If it matches the real penalty, then the measured variances carry no information and only the shrinkage matters. On LIDC-IDRI, the deep-learning context is DenseSharp+ at 91.52% accuracy and 0.9566 AUC on the high-ambiguity cohort (Probabilistic Radiomics). That is a different model class, and we cite it for context rather than as a target.

Beating the baselines means the following. Averaged over 50 repeated stratified splits, and with paired bootstrap intervals, at every test reliability level between 0.75 and 0.85 the variance-penalized curve must sit above the rerun ICC filter, ridge, and the permuted control by a margin whose interval excludes zero. A test set of 90 patients alone gives C-index intervals far too wide for single-split claims, which is why we use repeated splits.

Why not just…?

Why not just keep the ICC filter and tune its threshold? Any per-feature threshold makes a keep-or-drop decision one feature at a time, and it never sees how errors combine within the risk score. Even the best threshold therefore cannot keep a fragile feature that the remaining features happen to stabilize. The published drop to 0.641 shows how expensive a single threshold can be. Our sweep over thresholds gives the filter its best chance, and if its curve matches ours, the claim fails.

Why not just use ridge regression, which already shrinks everything? Ridge shrinks a robust shape feature as hard as a fragile texture feature. If fragility and predictiveness were unrelated, ridge would do as well as our method. The perturbation study's finding that predictive features tend to be fragile is evidence that they are related, and the matched-path comparison against ridge, together with the permuted-variance control, tests this directly.

Why not just train on the perturbed copies as data augmentation? This alternative is a real threat. For squared-error loss, averaging the loss over noisy copies equals the clean loss plus $n\,\beta^\top\bar\Sigma\beta$, so augmentation implicitly applies the same full-covariance penalty. For the Cox likelihood the correspondence holds only approximately, to second order. We expect augmentation and the explicit penalty to land close together on the main curve, and we will report that honestly. The explicit form would still contribute three things augmentation does not. It costs one fit instead of a fit on a 60-times-larger dataset. It exposes $\bar\Sigma$ for inspecting which features cancel. And it yields the per-patient variance $v_i$ for the patient weighting and test-time flagging. If augmentation also matches the patient-weighted variant on both C-index and selective-prediction gains, then the contribution reduces to a diagnostic tool, and we will say so.

Why not just use an end-to-end 3D network? On whole-body MRI, tabular radiomics beat a 3D ResNet18 (AI-driven preclinical disease risk assessment using imaging in UK biobank). Survival cohorts of a few hundred patients also favor low-dimensional models. Deep models remain out of scope here rather than ruled out.

Decisive Checks

The proposal stands or falls on one plot: the curve of test C-index against test reliability on Head-Neck-PET-CT, comparing the full-covariance penalty with the ICC-threshold curve, ridge, augmentation, and the permuted control. If, at a reliability ICC of about 0.8, the variance penalty reaches a C-index meaningfully above the rerun filter's value, and the permuted control does not, then measured perturbation variance is useful information that hard filtering throws away. If ridge or the permuted control matches it, then the reliability–accuracy trade-off is a matter of shrinkage strength rather than of knowing which features are fragile.

The LIDC-IDRI replication with real reader contours tests whether this holds beyond synthetic perturbations. The binned analysis of $v_i$ tests whether per-patient variance measures something clinically usable: if C-index on the most stable 80% of test patients does not exceed C-index on a random 80%, then the patient-level component has failed.