Revisiting Clinical Outcome Prediction for MIMIC-IV

Tom Röhr, Alexei Figueroa, Jens-Michalis Papaioannou, Conor Fallon, Keno Bressem, Wolfgang Nejdl, Alexander LöserNAACL 2024

TL;DR

This paper adapts existing clinical outcome prediction (COP) tasks to the newer MIMIC-IV dataset, which includes more diverse patient data beyond intensive care units. They introduce a novel patient routing task and find that PubMedBERT, with its domain-specific tokenizer, significantly outperforms other models, highlighting the importance of specialized pre-training for clinical NLP.

What Problem Does This Paper Address?

Clinical Decision Support Systems (CDSS) are crucial for helping medical professionals provide optimal patient care. A common approach to building these systems involves training models on electronic health records (EHRs), with the MIMIC dataset being a popular choice. However, most prior work has focused on MIMIC-III. This paper addresses the need to update these prediction tasks for MIMIC-IV, the latest version of the dataset, which offers a larger and more diverse patient population, including those from emergency departments. This update is essential because MIMIC-IV's expanded scope and updated coding standards (ICD-10) make previous MIMIC-III-based tasks potentially obsolete. The authors also highlight the challenge of class imbalance in clinical data, where many rare conditions or outcomes are underrepresented, making accurate prediction difficult.

Methods

The core of this work involves adapting existing clinical outcome prediction (COP) tasks from MIMIC-III to MIMIC-IV. This includes updating the data processing methods and, importantly, transitioning from ICD-9 to ICD-10 codes for diagnoses and procedures. A novel task, "patient routing," is introduced, which aims to predict the initial hospital unit a patient will be transferred to upon admission, leveraging MIMIC-IV's specific routing information.

The authors evaluate several bio-medical transformer encoder models: BioBERT, CORe, ClinicalBERT, DischargeBERT, and PubMedBERT. They fine-tune these models on the MIMIC-IV datasets for various tasks and report performance using AUROC (Area Under the Receiver Operating Characteristic curve) and PR-AUC (Area Under the Precision-Recall Curve). PR-AUC is emphasized as a more balanced metric for imbalanced datasets. Hyperparameter optimization (learning rate and warmup steps) was performed for PubMedBERT.

Datasets

The primary dataset used is MIMIC-IV, the latest version of the Medical Information Mart for Intensive Care database. This dataset includes EHRs from both intensive care units (ICUs) and emergency departments (EDs).

The paper also references MIMIC-III as the basis for previous COP tasks (COP-III). The authors created COP-IV, an updated and extended set of 6 clinical outcome prediction tasks based on MIMIC-IV. This includes:

  • 3 tasks adapted from COP-III (diagnoses, procedures, length-of-stay) for the MIMIC-IV ICU split.
  • An additional split for the ED, including diagnoses and procedures.
  • A novel patient routing task for the ED split.

Key Insights and Ideas

  • MIMIC-IV is a necessary update: The expanded scope of MIMIC-IV (including ED data) and the shift to ICD-10 codes necessitate re-evaluating and updating existing clinical prediction tasks.
  • Domain-specific tokenizers matter: PubMedBERT, which uses a domain-specific tokenizer and was pre-trained on biomedical literature, significantly outperforms other models. This suggests that specialized vocabulary and context are more critical than simply pre-training on MIMIC-III data itself.
  • Long-tail challenges persist: The PR-AUC metric reveals that while models perform well on common outcomes (the "head" of the distribution), they struggle with rare conditions (the "tail"). This class imbalance remains a significant hurdle in clinical NLP.
  • ICD-10 increases complexity: The transition to ICD-10 codes, which have a much larger label space than ICD-9, makes tasks like diagnoses and procedures prediction more challenging due to increased class imbalance.
  • Patient routing is a novel and useful task: The introduction of patient routing offers a new dimension to clinical outcome prediction, with potential applications in resource allocation and patient flow management.

Key Results

  • PubMedBERT Dominance: PubMedBERT consistently achieved the highest AUROC and PR-AUC scores across most tasks, often by a significant margin. For example, in the ED split for diagnoses prediction, PubMedBERT achieved an AUROC of 86.86% and a PR-AUC of 17.24%, compared to other models which had PR-AUCs around 14-15%.
  • PR-AUC Gap: A substantial gap between AUROC and PR-AUC was observed for most tasks, particularly for diagnoses and procedures prediction, underscoring the challenge of predicting rare events.
  • MIMIC-III pre-training offers limited benefit: Models pre-trained on MIMIC-III (CORe, ClinicalBERT, DischargeBERT) did not show a significant performance advantage over BioBERT (pre-trained on PubMed) when adapted to MIMIC-IV, suggesting that the domain-specific tokenizer of PubMedBERT was a more impactful factor.
  • Patient Routing Performance: All models achieved high AUROC scores for patient routing, but PR-AUC scores indicated difficulties in predicting less frequent hospital unit transfers.
  • Demographic Impacts: For patient routing, predictions were worse for male patients, and physician referrals led to better routing predictions, suggesting that the specificity of the admission note is key.

Limitations and Future Work

  • Class Imbalance: The paper repeatedly highlights the significant challenge posed by the long-tail distribution of labels in clinical data, which severely impacts PR-AUC performance. Future work should focus on developing models specifically designed to handle this imbalance.
  • Label Inconsistency: MIMIC datasets, while valuable, suffer from annotation inconsistencies. Future research could explore data augmentation techniques to improve label quality.
  • Dataset Generalization: Most prior work has focused on MIMIC. Evaluating models on datasets from different clinics, specialties, and languages is crucial to assess generalizability and mitigate potential overfitting.
  • Multimodal Data: The authors suggest enriching textual representations with multimodal data (e.g., imaging, lab results) could further improve prediction performance.
  • Novel Tasks: The paper proposes that exploring additional novel outcome prediction tasks would provide deeper insights into model capabilities in real-world clinical settings.

Summary updated

BibTeX
@inproceedings{rohr2024revisiting,
  author = {Tom Röhr and Alexei Figueroa and Jens-Michalis Papaioannou and Conor Fallon and Keno Bressem and Wolfgang Nejdl and Alexander Löser},
  title = {{Revisiting Clinical Outcome Prediction for MIMIC-IV}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure