Large-scale benchmark study of survival prediction methods using multi-omics data

Moritz Herrmann, Philipp Probst, Roman Hornung, Vindi Jurinović, Anne‐Laure BoulesteixBriefings 2020 · DOI

Abstract

Multi-omics data, which integrate diverse high-dimensional molecular variables, are increasingly used to investigate disease outcomes. However, their utility for predicting survival time remains a subject of debate. This study presents a large-scale benchmark using 18 cancer datasets from The Cancer Genome Atlas (TCGA), comparing 11 machine learning and statistical methods—including boosting, penalized regression, and random forests. The study evaluates whether methods that account for the group structure of omics data outperform traditional clinical models. Results indicate that while structured methods slightly improve performance by protecting clinical variables from molecular noise, the overall utility of multi-omics data for survival prediction in these datasets remains limited. Only the block forest method marginally outperformed a standard Cox model on average.

Introduction

In the era of precision medicine, researchers have access to unprecedented volumes of biological data. For a single cancer patient, multi-omics profiles can include genomic mutations, gene expressions (RNA), copy number variations (CNV), and microRNA. The theoretical promise is that by integrating every molecular layer, we can predict patient survival with high precision.

However, this potential is often hindered by the "curse of dimensionality." In a typical TCGA dataset, the number of features ($p$) can exceed 100,000, while the number of observations ($n$) ranges from only 35 to 1,000. This $p \gg n$ scenario causes standard statistical models to fail. Furthermore, multi-omics data is inherently structured; features belong to specific biological layers. This study provides a rigorous benchmark to determine if complex models can truly outperform simple clinical variables like age and tumor stage.

Methodology

Data Sources and Processing

The benchmark utilized 18 different cancer types from The Cancer Genome Atlas (TCGA). For each cancer, five distinct data groups were merged:

  • Clinical: Age, sex, and tumor stage (typically < 20 features).
  • CNV: Copy number variations (~60,000 features).
  • RNA: Gene expression (~20,000 features).
  • Mutation: Somatic mutations (~15,000 features).
  • miRNA: microRNA expression (~500–1,000 features).

Prediction Methods

Eleven methods were compared, categorized by their approach to the data's group structure:

  1. Lasso-based methods: Standard Lasso, IPF-Lasso (using different penalty factors per group), and Priority-Lasso (fitting groups successively based on priority).
  2. Statistical Boosting: Model-based boosting (mboost) and Likelihood-based boosting (CoxBoost).
  3. Random Forest: Standard implementations (rfsrc and ranger) and the structured Block Forest.

To handle the massive feature count, the study implemented "Two-Step" versions of IPF-Lasso and Priority-Lasso. These use a preliminary ridge regression to estimate group importance, making these methods computationally feasible for datasets with 100,000+ variables.

Evaluation Metrics

Performance was assessed using several repetitions of 5-fold cross-validation. The primary metrics were Uno’s C-index (measuring discriminative ability) and the integrated Brier score (measuring prediction accuracy).

Key Insights

The Clinical Anchor

Clinical variables are remarkably dense with information. A simple Cox model using only clinical data served as a high-performance baseline. The study found that "naive" machine learning methods—those that treat all 100,000+ features as a flat list—often performed worse than the clinical baseline. In these naive models, the predictive signal from clinical variables is frequently drowned out by molecular noise.

The Necessity of Structure

Methods that explicitly account for the multi-omics structure performed better. The most successful complex models were those that "favored" clinical data, including them without penalization and using molecular data only to explain residual variation. This approach ensures that known prognostic factors are not lost in the high-dimensional search space.

  • No Universal Winner: Across the 18 cancer types, no complex multi-omics method consistently and significantly outperformed the clinical-only Cox model.
  • Block Forest and CoxBoost: Block forest was the only method to outperform the Cox model on average, though only slightly. CoxBoost (when favoring clinical variables) also proved highly competitive.
  • The Sparsity Paradox: IPF-Lasso produced the most minimalist models by selecting the fewest features, while Priority-Lasso was more maximalist. Interestingly, the largest data group (CNV) was often the least useful for prediction.
  • Cancer-Specific Success: While the average performance was flat, molecular data added significant value in specific cases, such as Low-Grade Glioma (LGG), where molecular markers are biologically known to be critical for prognosis.

Conclusion and Future Work

The study serves as a sobering reality check for multi-omics survival prediction. In many current TCGA studies, the added value of high-dimensional molecular data over simple clinical variables is marginal.

Future research should focus on:

  1. Computational Efficiency: Many promising methods, such as Sparse Group Lasso (SGL), remain computationally prohibitive for 100,000+ features.
  2. Structured Random Forests: Developing more sophisticated versions of Random Forest that can prioritize clinical data while capturing non-linear molecular effects.
  3. Sample Size: Increasing the number of observations to better match the dimensionality of the features.

Ultimately, for multi-omics to deliver on its promise, models must be designed to respect the hierarchy of biological information, treating clinical data as the foundation rather than just another set of variables.

Summary updated

Abstract

Multi-omics data, that is, datasets containing different types of high-dimensional molecular variables, are increasingly often generated for the investigation of various diseases. Nevertheless, questions remain regarding the usefulness of multi-omics data for the prediction of disease outcomes such as survival time. It is also unclear which methods are most appropriate to derive such prediction models. We aim to give some answers to these questions through a large-scale benchmark study using real data. Different prediction methods from machine learning and statistics were applied on 18 multi-omics cancer datasets (35 to 1000 observations, up to 100 000 variables) from the database 'The Cancer Genome Atlas' (TCGA). The considered outcome was the (censored) survival time. Eleven methods based on boosting, penalized regression and random forest were compared, comprising both methods that do and that do not take the group structure of the omics variables into account. The Kaplan-Meier estimate and a Cox model using only clinical variables were used as reference methods. The methods were compared using several repetitions of 5-fold cross-validation. Uno's C-index and the integrated Brier score served as performance metrics. The results indicate that methods taking into account the multi-omics structure have a slightly better prediction performance. Taking this structure into account can protect the predictive information in low-dimensional groups-especially clinical variables-from not being exploited during prediction. Moreover, only the block forest method outperformed the Cox model on average, and only slightly. This indicates, as a by-product of our study, that in the considered TCGA studies the utility of multi-omics data for prediction purposes was limited. Contact:[email protected], +49 89 2180 3198 Supplementary information: Supplementary data are available at Briefings in Bioinformatics online. All analyses are reproducible using R code freely available on Github.
BibTeX
@inproceedings{herrmann2020large,
  author = {Moritz Herrmann and Philipp Probst and Roman Hornung and Vindi Jurinović and Anne‐Laure Boulesteix},
  title = {{Large-scale benchmark study of survival prediction methods using multi-omics data}},
  booktitle = {Briefings},
  year = {2020},
  doi = {10.1093/bib/bbaa167},
}
Figure