What Survival Benchmarks Don't Tell You: Impact of Model Selection and Dataset Regimes

Sandra Dening, Emre Kavak, Christian Wachinger, George H. ChenarXiv 2026 · arXiv

TL;DR

Survival analysis benchmarking has long suffered from a "C-index tunnel vision," leading researchers to believe that classical Cox models are still the unbeatable gold standard. This paper conducts a massive benchmark across 49 datasets to show that by simply changing the model selection criteria and accounting for dataset scale, modern deep learning and tree-based methods reveal significant advantages in probabilistic accuracy and calibration that the C-index completely misses.

What Problem Does This Paper Address?

Survival analysis—predicting the time until an event occurs (like a machine failure or a patient's relapse)—is a cornerstone of medical and engineering research. However, the field has hit a bit of a plateau in its evaluation practices.

Most studies rely almost exclusively on the Concordance Index (C-index), which measures how well a model ranks individuals by risk. The authors argue this is a narrow view. A model might be great at ranking patients but terrible at providing accurate probabilities or precise time-to-event estimates. Furthermore, many existing benchmarks exclude deep learning (DL) methods or focus on small, low-dimensional datasets, leading to the consensus that "Cox is all you need." This paper aims to break that consensus by treating evaluation design itself as an experimental variable.

Methods

The researchers conducted a large-scale, neutral benchmark comparing 10 methods (ranging from classical Cox models to tree-based ensembles like XGBoost and complex DL architectures like SurvTRACE) across 49 diverse datasets.

The key innovation is the Triple Model Selection Criteria. Instead of just tuning hyperparameters to maximize the C-index, they tuned each model three separate times to optimize for:

  1. Antolini C-index ($C^{td}$): Discriminative ranking.
  2. Integrated Brier Score (IBS): Probabilistic accuracy (a "proper scoring rule").
  3. Mean Absolute Error via Pseudo-Observations (MAE-PO): Time-to-event accuracy.

To make sense of the results, they used Bayesian Generalized Linear Mixed-effects Models (GLMMs). This allowed them to disentangle how much of a model's performance was due to the algorithm itself versus the inherent difficulty of the dataset.

Figure 3.1: Benchmark setup overview.

Datasets

The study utilized 49 datasets spanning clinical, omics, engineering, socio-economics, and finance domains.

  • Scale: Sample sizes ranged from tiny ($N=26$) to large ($N=100,000$).
  • Dimensionality: Feature counts ranged from 3 to 15,500.
  • Censoring: Rates varied from 0% to over 95%.
  • Sources: Included the SurvSet repository and large-scale external datasets like KKBox (music streaming churn), MIMIC-IV (ICU mortality), and METABRIC (breast cancer).

Key Insights and Ideas

  • The C-index is a Dataset Property, Not a Model Property: The Bayesian analysis revealed that 67.4% of the variance in C-index scores is explained by the dataset's baseline difficulty, while the choice of algorithm only accounts for a small fraction. If you only look at the C-index, all models look roughly the same.
  • The "Free Lunch" of IBS Optimization: For complex models like XGBoost and DeepSurv, selecting hyperparameters to optimize for IBS instead of C-index yields massive gains in probabilistic accuracy and calibration with almost zero cost to ranking performance.
  • The Calibration Fallacy: The authors discovered that high censoring rates artificially inflate calibration scores. Models appear "well-calibrated" simply because the metric is evaluated on highly censored data, masking underlying miscalibration.
  • DL Scaling vs. Dimensionality: While DL models scale exceptionally well as sample size ($N$) increases, they suffer significant performance drops in high-dimensional ($P > N$) spaces, suggesting they lack the necessary internal regularization for "omics" style data.
Figure 4.1: Bayesian Variance Partitioning Across Evaluation Metrics with 94% highest-density interval. † denotes log scale.

Key Results

The paper provides a detailed breakdown of how switching selection targets impacts performance across different architectures:

  • Time-to-Event Accuracy: When models were tuned for C-index or IBS, they only beat a simple Kaplan-Meier baseline 50% of the time. When tuned specifically for MAE-PO, this success rate jumped to 72%.
  • XGBoost Gains: Switching from C-index to IBS optimization for XGBoost resulted in a substantial median IBS improvement of -0.083 ($pd = 1.0$), proving that standard tuning practices leave a lot of performance on the table.
  • Deep Learning Failures: In high-dimensional regimes, models like DeepHit and SurvTRACE showed significant IBS deterioration (posterior effects of +0.026 and +0.02), highlighting a structural vulnerability to many features.
Table 4.1: Effects of model selection strategy on target metrics, when switching selection strategy From →To. Insignificant results (pd < 0.90) are kept colorless. Significant results are color-coded (light green for beneficial, light red for detrimental) and marked with asterisks (∗pd ≥0.90, ∗∗pd ≥0.95, ∗∗∗pd ≥0.99). A dagger (†) denotes metrics that are reported on a log-multiplier scale.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Skew: Despite the large number of datasets, the corpus is still skewed toward small-to-mid-sized clinical data.
  • Compute Constraints: A fixed CPU-only environment and a 15-hour compute budget per experiment may have disadvantaged deep learning models that thrive on GPUs and longer training.
  • Scope: The benchmark is restricted to right-censored, single-event, static tabular data, leaving out competing risks and longitudinal data.

Future Work: The authors advocate for "regime-conditioned reporting"—moving away from single-number summaries and toward transparently reporting how models perform under specific dataset characteristics and optimization targets. They also call for new architectures that structurally couple calibration with temporal precision.

Summary updated

Abstract

Despite the proliferation of survival analysis methods, current benchmarks often operate under narrow conditions: they heavily rely on the concordance index (C-index) for evaluation and model selection, and frequently either exclude deep learning (DL) methods or underrepresent challenging dataset regimes. To address these limitations, we conduct a large-scale, neutral benchmark comparing 10 classical, tree-based, and DL methods across 49 diverse datasets. Crucially, we treat the model selection criterion (optimizing for C-index, Integrated Brier Score (IBS), or Mean Absolute Error (MAE-PO)) as an independent experimental variable. Using Bayesian mixed-effects modeling, we demonstrate that evaluation design heavily dictates conclusions. For complex DL and tree-based architectures, selecting hyperparameters to optimize for IBS instead of C-index yields substantial improvements in probabilistic accuracy and calibration at a negligible cost to discriminative ranking. Furthermore, method performance (C-index, IBS, MAE-PO, and X-CAL) is fundamentally governed by dataset characteristics. While DL models scale exceptionally well with larger sample sizes, they suffer notable performance drops in high-dimensional feature spaces. Additionally, we expose a structural vulnerability where heavy censoring artificially inflates absolute calibration metrics, creating overly conservative tests. Ultimately, our findings challenge the utility of global single-number performance summaries and advocate for survival analysis results that transparently report the evaluation setup. Our code is available at: https://github.com/sdening/surv-bench-study
BibTeX
@misc{dening2026what,
  author = {Sandra Dening and Emre Kavak and Christian Wachinger and George H. Chen},
  title = {{What Survival Benchmarks Don't Tell You: Impact of Model Selection and Dataset Regimes}},
  year = {2026},
  eprint = {2611.24837},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24837},
}
Figure