Hundreds of cancer papers report "prognostic signatures": short lists of genes whose weighted expression predicts patient survival. Each is produced by the same recipe. Researchers screen genes one at a time for association with survival, fit a LASSO Cox model on the survivors, and report how well the resulting risk score separates patients. A LASSO Cox model is a survival regression that shrinks most gene weights to exactly zero. The recipe looks deterministic, but it is not. The screening cutoff, the rule for picking the penalty strength, the random assignment of cross-validation folds, the minimum follow-up filter and the train/validation split are all free choices. Changing any of them can change which genes come out and how good the signature looks. In this respect the pipeline behaves exactly like a metaheuristic, a stochastic search algorithm such as a genetic algorithm. For metaheuristics, a whole literature exists on how sensitive performance is to configuration settings and how to model that sensitivity cheaply.
We propose to bring that literature's tools to bear on survival signatures. We treat the signature pipeline as a configurable stochastic algorithm and fit empirical performance models over its configuration space. An empirical performance model is a surrogate regression that predicts how well an algorithm will do from its settings. With these models we measure how much of a signature's reported accuracy and gene content comes from substantive choices versus arbitrary ones, such as fold seeds. We then test a configuration-selection rule that optimizes for robustness rather than for the best single run. The intended outcome is a quantified answer to how much published signature performance is a product of tuning, plus a selection procedure whose internal estimates actually predict performance in external cohorts.
Core Claim
Metaheuristic research has shown that good settings are instance-dependent and fail to transfer. Settings tuned on one training instance degrade on new ones (Yuan Yuan et al.). Small perturbations of parameters can trap solvers (Feng Yao et al.). Brute-force tuning is too expensive because each configuration must be run many times to average out randomness (Ke Li 0001 et al.).
Our claim is that LASSO Cox signature pipelines show the same three properties:
- Instance dependence. The best configuration found on one cohort does not transfer to another.
- Configuration sensitivity. Small changes in screening threshold or penalty rule change the selected genes.
- Hidden selection. Implicitly choosing among many configurations inflates reported validation numbers.
We further claim that two specific tools from performance modeling make this measurable and partly correctable. The first is a rank-preserving surrogate over configurations. The second is a robustness-penalized selection criterion.
The specific novelty is twofold. First, a variance decomposition of signature performance and gene membership across pipeline configurations, done on real multi-cohort oncology data. Second, an external-cohort test of whether internal cross-validation rankings of configurations agree with external rankings at all.
The prior work plays three roles here.
- Method. Which Surrogate Works for Empirical Performance Modelling? A Case Study with Differential Evolution modeled differential evolution's population size, mutation scale and crossover rate on synthetic benchmarks. Each of 5,940 configurations was run 31 times. Gaussian processes and random forests gave the best point accuracy. Support vector regression had poor point error but competitive rank correlation. We reuse this design, including its emphasis on rank fidelity over pointwise error. We apply it to a small configuration space of a statistical pipeline rather than to an optimizer on synthetic functions. That sidesteps the high-dimensional surrogate failure the paper reports.
- Analogy for cheap evaluation. Tuning Local Search for Satisfiability Testing introduced retrospective parameter variation. It reuses traces from one long run to simulate many smaller cutoff settings at no extra cost. The LASSO regularization path has the same property: one fit yields results for every penalty value. We exploit this to make dense configuration sweeps affordable.
- Motivation. The signature papers supply the pipelines we audit. These include the 15-lncRNA breast cancer model (An exosome-related long non-coding RNAs risk model could predict survival outcomes in patients with breast cancer) and the 5-gene glioblastoma ferroptosis model (Ferroptosis-Related Gene Contributes to Immunity, Stemness and Predicts Prognosis in Glioblastoma Multiforme). Each reports a single configuration's outcome and does not examine how that outcome varies across configurations.
- Theory, which we complement. Oracle inequalities for the lasso in the Cox model gives theoretical error bounds for the Cox lasso estimator itself. It says nothing about the screening, filtering and penalty-selection choices wrapped around the estimator in practice, and those choices are what we study.
Research Program
Defining the configuration space. We first formalize the pipeline as a function $A(\theta, s, D)$. Here $\theta$ is a configuration, $s$ is a random seed governing fold and split assignment, and $D$ is a cohort. The configuration $\theta$ has roughly eight dimensions:
- the univariate screening cutoff (none, or a p-value threshold);
- the candidate gene universe (a curated biological set versus genome-wide);
- the minimum follow-up filter (for example, the 90-day rule used for TCGA-BRCA);
- the penalty-selection rule (the cross-validation minimum versus the more conservative one-standard-error rule);
- the cross-validation criterion (partial-likelihood deviance versus concordance);
- the number of folds;
- the elastic-net mixing weight, which blends the LASSO penalty with a ridge penalty that keeps correlated genes together;
- the expression normalization.
Each run returns three things: an internal estimate (the cross-validated C-index), an external estimate (C-index and time-dependent AUC on independent cohorts), and the selected gene set. The C-index is the fraction of patient pairs whose predicted risks are ordered consistently with their observed survival.
Sweeping and decomposing. Following the differential evolution study, we sample about 2,000 configurations per cohort and run each with 31 seeds. Because one path fit covers all penalty values, the penalty dimension adds almost no cost. We fit random-forest and Gaussian-process surrogates from $\theta$ to internal performance. Using the random-forest surrogate's functional analysis of variance, we split total variance into a configuration part, a seed part and their interaction. We do the same for gene-set membership, measured as pairwise Jaccard overlap between selected sets. The headline quantity is the share of external-performance variance explained by $\theta$ versus by $s$. If seeds dominate, the "optimized" configurations in published papers are partly lottery tickets.
A robust selection rule. Second, we test a selection rule borrowed from the robustness view of parameter tuning. The standard pipeline picks the configuration with the best single internal score. We instead pick
$$\theta^{*} = \arg\max_{\theta} \; \hat{\mu}(\theta) - \kappa\, \hat{\sigma}(\theta),$$
where $\hat{\mu}(\theta)$ and $\hat{\sigma}(\theta)$ are the surrogate-smoothed mean and seed-to-seed standard deviation of internal C-index, and $\kappa$ is a fixed risk-aversion weight. We set $\kappa = 1$ in advance and report sensitivity to it, without tuning it on external data. The final signature is refit at $\theta^{*}$, with genes retained only if they are selected in a majority of seeds.
Baselines
Naive reference. The do-nothing reference is a Cox model on available clinical covariates (age, stage, and subtype where available) with no gene signature. A signature that does not beat it externally adds nothing.
Standard practice. The main practical baseline is the published pipeline itself, re-run exactly as described: univariate screening followed by LASSO Cox with cross-validation-minimum penalty, on the same cohorts and splits. The reported numbers we must reproduce or explain are:
- TCGA-BRCA lncRNA model: time-dependent AUC of 0.750 to 0.798 at 2, 3 and 5 years (An exosome-related long non-coding RNAs risk model could predict survival outcomes in patients with breast cancer).
- Glioblastoma model: 3-year AUC of 0.706 and 5-year AUC of 0.782 (Ferroptosis-Related Gene Contributes to Immunity, Stemness and Predicts Prognosis in Glioblastoma Multiforme).
- ccRCC, where the best of 101 combinations (LASSO plus survival support vector machine) reached a C-index of 0.650 (Exploring the Prognostic Value of Tumour‐Associated Genes in Clear Cell Renal Cell Carcinoma Through Single‐Cell RNA Sequencing Insights). That paper is itself a case of selecting the best of many configurations.
Strongest statistical competitor. This is nested cross-validation over the same configuration grid, with an equal compute budget: the same number of pipeline evaluations, the same cohorts, and the same external test sets, which no method touches during selection. On METABRIC, the reference point is the linear Cox C-index of 0.816 reported in DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. That paper's DeepSurv reached 0.867. We include DeepSurv only as a ceiling for non-sparse models, not as a competitor for an interpretable signature.
Ablations. Two ablations isolate the key ingredient. The first uses the robust rule without surrogate smoothing, taking raw seed means and deviations. The second uses surrogate smoothing with $\kappa = 0$. Together they separate the benefit of variance penalization from the benefit of the surrogate.
What counts as beating the baselines. The robust rule beats standard practice if two conditions hold together: (1) its external C-index is at least as high, and (2) its gene sets are more stable across seeds and bootstrap resamples. Specifically, the median Jaccard overlap must be higher by a margin exceeding bootstrap noise, and the gap between internal and external performance must be smaller. Matching nested cross-validation on external accuracy while giving more stable gene sets also counts as a win, since nested cross-validation has no stability term.
Why not just…?
Why not just use nested cross-validation? Nested cross-validation gives an honest estimate of performance for the chosen procedure, and it is the real threat on the accuracy axis. If it matches the robust rule on external C-index, the predictive half of our claim fails. In that case the contribution reduces to the sensitivity audit. That audit still answers something nested cross-validation does not: how much of the variance is configuration versus seed, and how unstable the gene list is. The head-to-head comparison on external cohorts settles which case holds.
Why not just use elastic net, or bootstrap the signature? The elastic-net penalty addresses LASSO's arbitrary choice among correlated genes. Bootstrapping exposes sampling variability. Neither removes the analyst's other free choices, which are what we study. We therefore make the elastic-net mixing weight a configuration dimension and report seed and resample variability jointly, so both alternatives appear inside the decomposition rather than beside it.
Why not just run a metaheuristic wrapper that maximizes C-index directly? This is the obvious combination of the two directions. It resembles binary feature-selection wrappers such as An improved equilibrium optimization algorithm for feature selection problem in network intrusion detection. That method evolves gene-subset bitstrings scored by a classifier, reaching 97.6% accuracy on intrusion data with plenty of labeled samples.
With a few hundred censored patients and thousands of genes, a direct C-index search is precisely the over-tuning we want to measure. We expect it to widen the gap between internal and external performance. We include it as an arm with the same evaluation budget. If it beat the robust rule externally, we would have been wrong about where the risk lies.
Why not just use random survival forests or deep survival models? These models do not yield the short weighted gene list that clinical assays require. Moreover, DeepSurv reported random survival forests at only 0.619 C-index on METABRIC (DeepSurv). They answer a different question than the reliability of sparse signatures.
Proposed Experiments
Cohorts. We use the cohort pairings from the file:
- TCGA-BRCA (828 patients) for training, with METABRIC as the external cohort;
- TCGA-GBM for training, with CGGA-325 and GSE13041 as external cohorts;
- TCGA ccRCC for training, with ICGC and E-MTAB-1980 as external cohorts.
Metrics. We record Harrell's C-index and 3- and 5-year time-dependent AUC for performance, and Jaccard overlap of selected genes for stability. For surrogate quality, we use Spearman rank correlation between surrogate-predicted and observed performance, following Which Surrogate Works for Empirical Performance Modelling? A Case Study with Differential Evolution.
Transfer check. The first decisive check asks whether internal rankings transfer. We compute the Spearman correlation between configurations' internal C-index and their external C-index. A correlation near zero means that choosing among configurations on internal data is noise, so any best-of-many reporting is inflated.
Null check. The second check is a permutation null. We shuffle survival labels and rerun the full sweep, then record how high the best-of-configurations internal AUC climbs. That gives a direct estimate of how much apparent signal the forking paths alone can manufacture. It is the survival analogue of over-exploitation in a metaheuristic.
Placement check. The third check locates each published configuration within our sweep's distributions. If a published configuration sits in the upper tail internally but near the median externally, its reported number is partly a selection artifact.
Robust-rule check. The fourth check compares the robust rule, its two ablations, nested cross-validation, the direct C-index wrapper and standard practice on held-out external cohorts. It uses an equal evaluation budget, with 200 bootstrap resamples of the external cohort for confidence intervals.
What failure looks like. If configuration explains most external variance, internal rankings transfer, and the robust rule gives no stability gain, then current practice is sounder than we suspect, and we would report that plainly.
References
- Iterative genetic improvement: Scaling stochastic program synthesis
- Earth Observation Satellites Scheduling Based on Decomposition Optimization Algorithm
- Which Surrogate Works for Empirical Performance Modelling? A Case Study with Differential Evolution
- Tuning Local Search for Satisfiability Testing
- An exosome-related long non-coding RNAs risk model could predict survival outcomes in patients with breast cancer
- Ferroptosis-Related Gene Contributes to Immunity, Stemness and Predicts Prognosis in Glioblastoma Multiforme
- Oracle inequalities for the lasso in the Cox model
- Exploring the Prognostic Value of Tumour‐Associated Genes in Clear Cell Renal Cell Carcinoma Through Single‐Cell <scp>RNA</scp> Sequencing Insights
- DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network
- An improved equilibrium optimization algorithm for feature selection problem in network intrusion detection