FAMEWS: a Fairness Auditing tool for Medical Early-Warning Systems

Marine Hoche, Olga Mineeva, M. Burger, Alessandro Blasimme, Gunnar Rätsch

TL;DR

This paper introduces FAMEWS, an open-source tool designed to audit the fairness of medical early-warning systems (EWS). FAMEWS goes beyond standard fairness metrics by incorporating clinical outcome analysis and bias source screening, crucial for ensuring ethical AI in healthcare.

What Problem Does This Paper Address?

Machine learning (ML) models are increasingly used in healthcare for tasks like early detection of critical conditions. However, these models can exhibit biases, leading to unfair health outcomes for certain patient groups. While regulations are emerging to ensure ethical AI, translating these principles into practical, statistically sound fairness audits for complex medical systems like EWS remains a significant challenge. Existing fairness tools often lack the specific considerations needed for time-series data and clinical contexts, such as event-based metrics and analysis of real-world consequences.

Methods

FAMEWS provides a flexible, multi-stage fairness auditing framework tailored for medical EWS. Its key components include:

  • Flexible Patient Grouping: Allows users to define cohorts based on demographics (sex, age) or clinical attributes (admission type, comorbidities), enabling the identification of biases in subgroups beyond standard protected attributes.
  • Comprehensive Analysis Stages:
    • Model Performance Comparison: Evaluates classical fairness metrics (recall, precision, etc.) and specialized event-based and prevalence-corrected metrics across cohorts.
  • Model Outcome Comparison: Analyzes the time gap between the first correct alarm and the actual event, crucial for EWS where timing is critical.
  • Bias Source Screening: Investigates potential sources of bias by examining:
  • Label Bias: Compares medical variables used to define proxy labels across cohorts.
  • Feature Importance Bias: Assesses if feature importance rankings differ significantly across cohorts, which could lead to models that perform poorly for certain groups.
  • Missing Data Bias: Analyzes how the intensity of medical variable measurements varies across cohorts and its impact on model performance.
  • Automated PDF Reporting: Generates comprehensive, stakeholder-friendly reports that summarize audit findings, facilitating communication between developers, clinicians, and regulators.
  • Statistical Rigor: Employs bootstrapping and Mann-Whitney U tests with Bonferroni correction to ensure the statistical robustness of the findings.

Datasets

The primary dataset used for experiments and demonstrating FAMEWS is the HiRID dataset (Faltys et al., 2021), a high time-resolution ICU dataset. The authors also mention testing the framework on other alarm systems (e.g., respiratory failure detection) and datasets like MIMIC-III. The benchmark models for early-detection of organ failure used in the study were developed by Yèche et al. (2021).

Key Insights and Ideas

  • Beyond Standard Metrics: Fairness in medical EWS requires more than just comparing standard classification metrics. Analyzing clinical outcomes (like the time gap between alarm and event) and potential sources of bias (like missing data patterns) is essential.
  • Context Matters: The definition of fairness and the relevant metrics are highly context-dependent, especially in time-series medical data where events have a temporal dimension.
  • Subgroups Beyond Demographics: Identifying biases requires looking at clinically relevant subgroups (e.g., by admission type) in addition to standard demographic groups.
  • Transparency for Stakeholders: A well-structured, comprehensive report is crucial for bridging the gap between technical fairness audits and practical decision-making by clinicians, developers, and regulators.
  • Missing Data is a Major Bias Source: The intensity and patterns of missing medical data can significantly impact model performance and introduce biases across different patient groups.

Key Results

The paper presents detailed fairness audits of ML models on the HiRID dataset, showcasing FAMEWS's capabilities. While specific quantitative results are presented in the extensive tables and figures within the report appendix, the key qualitative findings include:

  • Performance Discrepancies: Significant performance differences were observed across various patient groupings (sex, age, APACHE group, surgical status) for different metrics. For instance, the study highlighted performance discrepancies for male patients and neurological patients.
  • Time Gap Disparities: The time gap between alarms and events varied significantly across cohorts, indicating potential fairness issues in how quickly different patient groups receive attention.
  • Medical Variable Distribution Differences: Distributions of key medical variables (like a_Lac and ABPm) differed notably across patient groups, suggesting potential label bias or differential measurement intensity.
  • Feature Importance Shifts: Feature rankings showed variations across cohorts, indicating that features important for the general population might not be as important for specific subgroups, potentially leading to reduced predictive power for those groups.
  • Impact of Missingness: Missingness in medical data was shown to have a significant impact on model performance, with certain metrics being substantially worse for patients with missing data.

Limitations and Future Work

  • Report Cumbersomeness: The audit report, while comprehensive, might appear cumbersome. The authors believe this is necessary for thoroughness, but acknowledge that selecting relevant results can be subjective.
  • Limited Scope of Bias Sources: The tool focuses on crucial bias sources but acknowledges that other factors might be relevant depending on the specific system.
  • Outcome Bias Measurement: Measuring outcome bias is application-dependent and might require post-deployment data.
  • Metric Selection: FAMEWS offers a curated set of metrics, including event-based and prevalence-corrected ones, but doesn't cover all possible fairness metrics.
  • Future Work: The authors encourage extending the fairness audit to post-deployment biases and exploring additional fairness metrics. They also suggest that users can adapt FAMEWS for other binary classifiers and extend its functionalities due to its open-source nature.

Summary updated

Abstract

Abstract Machine learning applications hold promise to aid clinicians in a wide range of clinical tasks, from diagnosis to prognosis, treatment, and patient monitoring. These potential applications are accompanied by a surge of ethical concerns surrounding the use of Machine Learning (ML) models in healthcare, especially regarding fairness and non-discrimination. While there is an increasing number of regulatory policies to ensure the ethical and safe integration of such systems, the translation from policies to practices remains an open challenge. Algorithmic frameworks, aiming to bridge this gap, should be tailored to the application to enable the translation from fundamental human-right principles into accurate statistical analysis, capturing the inherent complexity and risks associated with the system. In this work, we propose a set of fairness impartial checks especially adapted to ML early-warning systems in the medical context, comprising on top of standard fairness metrics, an analysis of clinical outcomes, and a screening of potential sources of bias in the pipeline. Our analysis is further fortified by the inclusion of event-based and prevalence-corrected metrics, as well as statistical tests to measure biases. Additionally, we emphasize the importance of considering subgroups beyond the conventional demographic attributes. Finally, to facilitate operationalization, we present an open-source tool FAMEWS to generate comprehensive fairness reports. These reports address the diverse needs and interests of the stakeholders involved in integrating ML into medical practice. The use of FAMEWS has the potential to reveal critical insights that might otherwise remain obscured. This can lead to improved model design, which in turn may translate into enhanced health outcomes.
BibTeX
@inproceedings{hoche2024famews,
  author = {Marine Hoche and Olga Mineeva and M. Burger and Alessandro Blasimme and Gunnar Rätsch},
  title = {{FAMEWS: a Fairness Auditing tool for Medical Early-Warning Systems}},
  booktitle = {Chil},
  year = {2024},
}
Figure