Expert-integrated automated machine learning uncovers hemodynamic predictors in spinal cord injury

Austin Chou, Abel Torres‐Espín, Nikos Kyritsis, J. Russell Huie, Sarah Khatry, Jeremy Funk, Jennifer Hay, Andrew Lofgreen, Rajiv Ratn Shah, Chandler McCann, Lisa U. Pascual, Edilberto Amorim, Philip R. Weinstein, Geoffrey A. Manley, Sanjay S. Dhall, Jonathan Z. Pan, Jacqueline C. Bresnahan, Michael S. Beattie, William D. Whetstone, Adam R. Ferguson, the Track-sci InvestigatorsbioRxiv 2021 · DOI

TL;DR

This paper introduces a novel framework that enhances Automated Machine Learning (AutoML) for biomedical applications by integrating explainability, reproducibility, and domain expertise. This approach was successfully applied to identify critical hemodynamic predictors for spinal cord injury (SCI) patient outcomes, demonstrating its potential to improve clinical decision-making and accelerate biomedical discovery.

What Problem Does This Paper Address?

The increasing power of AutoML in generating high-performing predictive models for healthcare is exciting, but a significant hurdle remains: translating these "black box" models into clinically actionable insights. Clinicians need to trust that models are not only accurate but also interpretable, verifiable, and reproducible. This is particularly challenging in biomedicine, where datasets can be small, and subtle changes in clinical practice can impact model performance over time. The authors tackle this by developing a framework that bridges the gap between automated model building and the need for clinical trust and understanding, specifically focusing on predicting outcomes in spinal cord injury patients.

Methods

The core of this work is a model-agnostic framework designed to augment standard AutoML pipelines. Key components include:

  • Reproducible AI Strategies:
    • Repeated k-fold cross-validation: Aggregating results from multiple cross-validation runs (25 in this study) to mitigate model instability due to data partitioning, leading to more robust performance estimates.
  • Performance Precision Analysis: Quantifying the reliability of model performance by measuring the width of confidence intervals relative to the mean performance.
  • Feature Instability Analysis: Measuring how much feature importance rankings shift across different data partitions, helping to identify stable and reliable predictors.
  • Explainable AI Tools:
    • Permutation Feature Importance (pFI): Identifying which features contribute most to model predictions.
  • Partial Dependence Plots (PDPs): Visualizing the marginal effect of individual features on the model's prediction.
  • Domain Expertise Integration: The framework explicitly incorporates clinical expertise at various stages, including feature selection and model verification, to ensure biological plausibility and clinical relevance.
  • Automated Feature Reduction: An iterative backward wrapper approach using pFI to identify and remove the least important features, leading to more parsimonious and potentially more generalizable models.
  • Population Similarity Analysis: Assessing data drift between training and validation cohorts to understand how evolving clinical practices might affect model performance.

Datasets

The study utilized two datasets from the Transforming Research and Clinical Knowledge for SCI (TRACK-SCI) program:

  • Training Dataset: 74 de-identified clinical records collected between 2005-2011.
  • Validation Dataset: 59 de-identified clinical records collected after 2015.

The prediction target was whether a patient's ASIA Impairment Scale (AIS) score improved between hospital admission and discharge.

Key Insights and Ideas

  • Hypertension is a Key Predictor: The study identified intraoperative hypertension (specifically, time spent above 104 mmHg) as a significant predictor of worse outcomes in SCI patients, even more so than hypotension. This challenges some traditional clinical focus and highlights the importance of precise blood pressure management during surgery.
  • Clinical Practices Evolve and Impact Models: The analysis revealed that shifts in clinical practices, such as faster surgical times and improved hypotension management, contributed to data drift between the training and validation sets. This underscores the critical need for ongoing model monitoring and re-validation.
  • Reproducibility is Achievable with AutoML: By augmenting standard AutoML with strategies like repeated cross-validation and performance precision metrics, the authors demonstrated that reproducible and interpretable models can be generated, even with smaller clinical datasets.
  • Domain Expertise Enhances Model Performance: Integrating clinical knowledge, such as preserving the hypotension-related feature time_MAP_Avg_below_76 during feature reduction, led to improved model performance in the BPXGB model.

Key Results

  • The developed framework successfully identified actionable intraoperative mean arterial pressure (MAP) thresholds associated with worse SCI patient outcomes.
  • The best-performing parsimonious models achieved high AUCs: BPlog reached an AUC of 0.84 ± 0.02, and BPXGB achieved 0.87 ± 0.01 after feature reduction.
  • Population similarity analysis revealed significant data drift for most features between the training and validation cohorts, indicating that evolving clinical practices had a notable impact.
  • While the BPXGB model performed slightly better on the training data, the BPlog model showed better performance on the external validation dataset, suggesting that simpler models or different feature sets might generalize better in certain scenarios.

Limitations and Future Work

  • Small Sample Size: The study acknowledges the challenge of small sample sizes in clinical datasets, which can lead to model instability. The framework's use of repeated cross-validation and aggregation helps mitigate this, but it remains a general limitation in SCI research.
  • Data Drift: The observed data drift highlights the need for continuous monitoring and re-validation of models as clinical practices evolve. Future work could focus on developing methods to automatically detect and adapt to such drifts.
  • Feature Collinearity: The paper notes that pFI can overestimate the importance of collinear features, particularly for tree-based models. Future research could explore alternative feature importance methods like SHAP or accumulated local effects plots.
  • Generalizability: While the framework is model-agnostic, its application to other clinical prediction tasks and broader patient populations would be valuable. The authors suggest that the framework can be applied to investigate other modifiable components of clinical care and predictors of chronic recovery.

Summary updated

Abstract

Abstract Automated machine learning (AutoML) is positioned to democratize artificial intelligence (AI) by reducing the amount of human input and ML expertise needed to create prediction models. However, successful translation of ML in biomedicine requires moving beyond optimizing only for prediction accuracy and towards discovering reproducible clinical and biological inferences. Here, we present a model-agnostic framework to reinforce AutoML using strategies and tools of explainable and reproducible AI, including novel metrics for performance precision and feature instability. The framework enables clinicians to interpret AutoML-generated models for clinical and biological verifiability and consequently integrate domain expertise during model development. We applied the framework towards spinal cord injury prognostication and identified a detrimental relationship between intraoperative hypertension and patient outcome. Furthermore, our analysis captured evolving clinical practices such as faster time-to-surgery and blood pressure management that affected clinical model validation. Altogether, we illustrate how augmenting AutoML for inferential reproducibility empowers biomedical discovery and builds trust in AI processes towards effective clinical integration.
BibTeX
@misc{chou2021expert,
  author = {Austin Chou and Abel Torres‐Espín and Nikos Kyritsis and J. Russell Huie and Sarah Khatry and Jeremy Funk and Jennifer Hay and Andrew Lofgreen and Rajiv Ratn Shah and Chandler McCann and Lisa U. Pascual and Edilberto Amorim and Philip R. Weinstein and Geoffrey A. Manley and Sanjay S. Dhall and Jonathan Z. Pan and Jacqueline C. Bresnahan and Michael S. Beattie and William D. Whetstone and Adam R. Ferguson and the TRACK-SCI Investigators},
  title = {{Expert-integrated automated machine learning uncovers hemodynamic predictors in spinal cord injury}},
  year = {2021},
  doi = {10.1101/2021.09.27.461544},
  howpublished = {bioRxiv},
}
Figure