Calibration and Uncertainty in Neural Time-to-Event Modeling

Paidamoyo Chapfuwa, Chenyang Tao, Chunyuan Li, Irfan Khan, Karen Chandross, Michael J. Pencina, Lawrence Carin, Ricardo HenaoIEEE TNNLS 2023 · DOI · OpenReview

Introduction

Survival analysis, or time-to-event modeling, is critical in domains ranging from healthcare to industrial maintenance. While traditional models like Cox Proportional Hazards (CPH) excel at ranking individuals by relative risk (measured by the Concordance Index or C-Index), they often fail to provide accurate absolute time predictions. This paper addresses the critical gap of calibration—ensuring that a model's predicted survival probabilities align with observed population outcomes.

The authors introduce Survival Function Matching (SFM), a novel neural approach that prioritizes calibration and sharpness without the instability typically associated with adversarial training.

Methodology

SFM is built on three core pillars designed to move beyond simple ranking to true distribution matching:

  1. Implicit Generative Modeling: Rather than assuming a fixed parametric distribution (e.g., Weibull), SFM uses a neural network $G_\theta(x, \epsilon)$ to transform noise $\epsilon$ into predicted event times. This allows the model to capture complex, multi-modal survival distributions.
  2. Differentiable Kaplan-Meier (DKM): The Kaplan-Meier estimator is the standard for empirical survival curves but is non-differentiable. The authors replace discrete indicators with differentiable Heaviside step functions, allowing the KM estimator to be integrated directly into the backpropagation loop.
  3. The SFM Objective: The model minimizes the distance between its predicted survival curve and the empirical ground truth. To maintain individual-level accuracy, it incorporates a supervised loss (Mean Absolute Error for events, hinge loss for censoring) similar to the DATE model.
Figure 1: Comparison of predicted survival curves across different models. SFM (orange) closely tracks the empirical survival curve (black) over time.

Unlike previous generative survival models that rely on adversarial discriminators, SFM achieves distribution matching through direct survival function alignment, leading to significantly more stable training.

Experimental Evaluation

The researchers evaluated SFM against five diverse datasets: EHR (394k+ clinical observations), FLCHAIN, SUPPORT, SEER (breast cancer), and SLEEP. These datasets represent a wide spectrum of censoring rates (23% to 68%) and temporal scales.

Key Results: Calibration and Sharpness

SFM demonstrates superior performance in two critical areas:

  • Population Calibration: In nearly every test case, SFM achieved a calibration slope closer to the ideal 1.0 than competitors like CPH, DRAFT, or S-CRPS. Visual evidence from calibration plots shows SFM following the diagonal line (perfect calibration) far more closely than other models.
  • Prediction Sharpness: A model can be calibrated by simply predicting the population average, but such predictions are not useful for individual decision-making. Using the Coefficient of Variation (CoV) as a metric for sharpness, the authors show that SFM produces concentrated, "sharp" distributions rather than smeared-out averages.
Figure 3: Calibration plots (left) and Coefficient of Variation (CoV) boxplots (right) for the SEER and SLEEP datasets. SFM shows high calibration (following the diagonal) and high sharpness (low CoV).

Discussion and Limitations

While SFM shows significant improvements in calibration, the authors identify challenges in specific scenarios. On the SEER dataset, performance was impacted by non-random "administrative censoring" at 80 months, which violates the model's assumption of non-informative censoring. Similarly, in the EHR dataset, the presence of non-i.i.d. data (multiple visits per patient) reduced calibration accuracy.

Future work will focus on extending SFM to handle informative missingness and non-independent clinical trajectories, further bridging the gap between theoretical survival modeling and real-world clinical application.

Summary updated

Abstract

Models for predicting the time of a future event are crucial for risk assessment, across a diverse range of applications. Existing time-to-event (survival) models have focused primarily on preserving pairwise ordering of estimated event times (i.e., relative risk). We propose neural time-to-event models that account for calibration and uncertainty while predicting accurate absolute event times. Specifically, an adversarial nonparametric model is introduced for estimating matched time-to-event distributions for probabilistically concentrated and accurate predictions. We also consider replacing the discriminator of the adversarial nonparametric model with a survival-function matching estimator that accounts for model calibration. The proposed estimator can be used as a means of estimating and comparing conditional survival distributions while accounting for the predictive uncertainty of probabilistic models. Extensive experiments show that the distribution matching methods outperform existing approaches in terms of both calibration and concentration of time-to-event distributions.
BibTeX
@article{chapfuwa2023calibration,
  author = {Paidamoyo Chapfuwa and Chenyang Tao and Chunyuan Li and Irfan Khan and Karen Chandross and Michael J. Pencina and Lawrence Carin and Ricardo Henao},
  title = {{Calibration and Uncertainty in Neural Time-to-Event Modeling}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2023},
  doi = {10.1109/tnnls.2020.3029631},
}
Figure