Flexible Group Fairness Metrics for Survival Analysis

Raphael Sonabend, Florian Pfisterer, Alan Mishler, Moritz Schauer, Lukas Burk, Sebastian J. Vollmer

Abstract

Survival analysis, which predicts the timing of events, is essential in high-stakes domains like healthcare and finance. However, standard fairness tools often fail to account for unique challenges such as censoring and distributional outputs. This paper evaluates a "Fairness through Loss" framework, repurposing existing survival metrics to detect algorithmic bias. By testing 29 datasets with injected permutation and undersampling bias, the researchers demonstrate that Integrated Brier Scores (RSBS) and Logloss (RISL) are highly effective, "strictly proper" metrics for auditing group fairness.

Introduction

While algorithmic fairness has gained significant traction in classification and regression, it remains underdeveloped in survival analysis. Survival models are critical for predicting patient mortality or loan defaults, yet they face two distinct hurdles: censoring (incomplete event data) and distributional outputs (predicting probabilities over time rather than binary outcomes). If these models are less accurate for specific demographic groups, they risk perpetuating systemic injustices. This study investigates which existing survival metrics can be effectively adapted to quantify such biases.

Methodology

The researchers employed a "Fairness through Loss" framework, defining fairness ($F_L$) as the absolute difference in performance between an advantaged and a disadvantaged group:

$$F_L = |L_{advantaged} - L_{disadvantaged}|$$

To evaluate metric sensitivity, the team utilized 29 public survival datasets and intentionally introduced bias through two methods:

  1. Permutation: Shuffling data for the disadvantaged group to simulate poor data quality.
  2. Undersampling: Reducing observations for the disadvantaged group to simulate representation bias.

Random Survival Forests were trained on these biased datasets, and the performance of various metrics—including Calibration, Discrimination, and Scoring Rules—was audited against the level of injected bias ($\sigma$).

Datasets

The study leveraged a diverse set of 29 publicly available survival datasets from R packages like survival and mlr3proba. These benchmarks include healthcare data (e.g., breast cancer, lung cancer, and stroke) and general datasets (e.g., veteran lung cancer). The datasets varied in size (137 to 52,000+ observations) and censoring rates (6% to 94%), ensuring a robust evaluation across different data conditions.

Results and Key Insights

The effectiveness of each metric was assessed using Spearman rank correlation to determine how well the fairness score ($F_L$) tracked with increasing bias ($\sigma$).

Table 1: Performance of fairness metrics under permutation and undersampling bias. The table displays regression coefficients (alpha, beta) and Spearman rank correlation (rho) for various metrics.

Discrimination vs. Calibration

The results indicated that while Harrell’s C-index ($C_H$) is excellent at detecting ranking errors (discrimination), it fails to capture calibration—how well predicted probabilities match reality. Conversely, some calibration metrics like "D-calibration" failed to detect biases that other metrics caught, suggesting that a model can appear calibrated on average while remaining unfair to specific subgroups.

The Superiority of Scoring Rules

Integrated Brier Scores (RSBS) and Logloss (RISL) emerged as the most comprehensive metrics. As "strictly proper" scoring rules, they are only maximized when the model predicts the true distribution, allowing them to capture both ranking and probability errors. As shown in Table 1, under permutation bias, $C_H$, $C_U$, RSBS, and RISL all showed nearly perfect correlation ($\rho \geq 0.976$). While undersampling bias was harder to detect, RSBS and RISL remained strong indicators of unfairness.

Discussion and Future Work

This research provides a model-agnostic framework for auditing fairness in survival analysis. However, several limitations remain. The study focused exclusively on right-censoring; future research should address left- and interval-censoring. Additionally, the bias models used (permutation and undersampling) are relatively coarse. Real-world bias may be more subtle, such as temporal bias where accuracy degrades only after a specific time point. By providing open-source code and a clear methodology, the authors aim to make fairness audits a standard component of the survival analysis pipeline.

Summary updated

BibTeX
@misc{sonabend2022flexible,
  author = {Raphael Sonabend and Florian Pfisterer and Alan Mishler and Moritz Schauer and Lukas Burk and Sebastian J. Vollmer},
  title = {{Flexible Group Fairness Metrics for Survival Analysis}},
  year = {2022},
  eprint = {2206.03256},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.03256},
}
Figure