Predicting cause of death from free-text health summaries: development of an interpretable machine learning tool

Chris McWilliams, Eleanor Walsh, Avon Huxor, Emma L. Turner, Raúl Santos‐RodríguezmedRxiv 2021 · DOI

Abstract

Accurately assigning cause of death is vital to understanding health outcomes and improving healthcare provision. While cancer-specific cause of death is a key outcome in clinical trials, assignment from death certification is prone to misattribution. We developed an interpretable machine learning classifier to predict prostate cancer death from free-text summaries of medical history for prostate cancer patients (CAP). Compared to independent human expert assignment, the classifier showed >90% accuracy. Visualizations were developed to highlight predictive elements, improving analyst confidence and revealing potential for decision-support tools.

Introduction

Determining the exact cause of death (CoD) is more complex than it appears. Death certificates, while official, are prone to misattribution—particularly in oncology, where patients may die with a disease rather than from it. In clinical trials, this noise can skew mortality outcome measures, often requiring expensive and time-consuming manual reviews by expert committees.

Researchers from the University of Bristol and the University of Exeter have developed a machine learning approach to automate this process using free-text medical summaries. A primary focus of this work is interpretability, ensuring that the "black box" of AI is transparent and useful for clinical decision-making.

Methods

The research team constructed a classification pipeline using a "bag-of-words" representation of health records. The process involved:

  1. Preprocessing: Text was lemmatized using the WordNetLemmatizer to reduce words to their base forms (e.g., "testing" to "test").
  2. Classification: Three standard algorithms were compared: Support Vector Classifiers (SVC), Logistic Regression (LR), and Random Forest (RF).
  3. Interpretability: Four methods—Gini Importance, TreeInterpreter, LIME, and SHAP—were used to quantify word-level contributions to predictions.
  4. Visualization: "Interpretable vignettes" were created, where font size and color in the original summaries indicate a word's influence on the prediction (e.g., blue for "prostate cancer death").

Dataset

The study utilized the CAP (Cluster randomised trial of PSA testing for prostate cancer) dataset, which includes 2,606 medical summaries of men diagnosed with prostate cancer who died by March 2016. Ground truth labels were established by an independent committee of urologists, pathologists, and oncologists.

Summary of patient characteristics and primary outcomes in the CAP dataset.

Results

All three classifiers demonstrated high performance. The Support Vector Classifier (SVC) achieved the highest Area Under the Curve (AUC) at 0.9749, while the Random Forest (RF) model provided a balanced performance with an accuracy of 91.8% and an F1 score of 0.931.

Performance metrics for the SVC and LR classifiers on the CAP dataset.

Key findings include:

  • Clinical Alignment: The model prioritized features like "bone scan," "spine," and "hormone," which are clinically recognized signs of progressing prostate cancer.
  • Contextual Nuance: The "interpretable vignettes" revealed that the absence of certain keywords could be as predictive as their presence.
  • Expert Trust: Preliminary feedback indicated that visualizing the model's reasoning significantly improved clinician confidence in the tool's potential as a decision-support aid.

Conclusion

The results suggest that an interpretability-first approach can produce high-accuracy tools that align with clinical expertise. While the current model struggles with negation and lacks numerical data (like PSA values), it represents a significant step toward automated, transparent cause-of-death assignment. The researchers have published their code on GitHub to facilitate further application in clinical settings.

Summary updated

Abstract

Structured Abstract Purpose Accurately assigning cause of death is vital to understanding health outcomes in the population and improving health care provision. Cancer-specific cause of death is a key outcome in clinical trials, but assignment of cause of death from death certification is prone to misattribution, therefore can have an impact on cancer-specific trial mortality outcome measures. Methods We developed an interpretable machine learning classifier to predict prostate cancer death from free-text summaries of medical history for prostate cancer patients (CAP). We developed visualisations to highlight the predictive elements of the free-text summaries. These were used by the project analysts to gain an insight of how the predictions were made. Results Compared to independent human expert assignment, the classifier showed >90% accuracy in predicting prostate cancer death in test subset of the CAP dataset. Informal feedback suggested that these visualisations would require adaptation to be useful to clinical experts when assessing the appropriateness of these ML predictions in a clinical setting. Notably, key features used by the classifier to predict prostate cancer death and emphasised in the visualisations, were considered to be clinically important signs of progressing prostate cancer based on prior knowledge of the dataset. Conclusion The results suggest that our interpretability approach improve analyst confidence in the tool, and reveal how the approach could be developed to produce a decision-support tool that would be useful to health care reviewers. As such, we have published the code on GitHub to allow others to apply our methodology to their data ( https://zenodo.org/badge/latestdoi/294910364 ).
BibTeX
@misc{mcwilliams2021predicting,
  author = {Chris McWilliams and Eleanor Walsh and Avon Huxor and Emma L. Turner and Raúl Santos‐Rodríguez},
  title = {{Predicting cause of death from free-text health summaries: development of an interpretable machine learning tool}},
  year = {2021},
  doi = {10.1101/2021.07.15.21260082},
  howpublished = {medRxiv},
}
Figure