Self-attention Enhanced Patient Journey Understanding in Healthcare System

Xueping Peng, Guodong Long, Tao Shen, Sen Wang, Jing Jiang

TL;DR

This paper introduces MusaNet, a novel multi-level self-attention network designed to understand complex patient journeys from electronic health records (EHRs). By effectively capturing both the contextual and temporal relationships within patient data, MusaNet achieves superior performance in predicting critical healthcare outcomes like re-admission and future diagnoses.

What Problem Does This Paper Address?

Understanding a patient's journey through the healthcare system is crucial for various applications, from predicting readmissions to improving medical research. This journey is inherently complex, structured across multiple levels: individual medical codes, visits to healthcare providers, and the overall sequence of these visits over time. The primary challenge lies in developing a model that can effectively encode these multi-level, temporal, and often irregular sequences of medical events. Existing methods, often relying on Recurrent Neural Networks (RNNs) or Convolutional Neural Networks (CNNs), struggle to fully capture the intricate dependencies and temporal gaps between patient visits, especially in long and complex medical histories.

Methods

The core innovation of this paper is the Masked Self-Attention (mSA) mechanism, which is designed to address the limitations of standard self-attention in capturing temporal relationships. mSA incorporates positional masks (forward, backward, and diagonal) to explicitly model the temporal order and dependencies between patient visits, even when they are separated by significant time intervals.

This mSA module is then integrated into a larger architecture called the Multi-level Self-Attention Network (MusaNet). MusaNet works as follows:

  1. Code-level Attention: Medical codes within each visit are first embedded and then processed by a code-level attention layer to generate visit embeddings.
  2. Interval Encoding: The temporal interval between consecutive visits is explicitly encoded and added to the visit embeddings.
  3. Masked Self-Attention (mSA): Two mSA blocks (one with a forward mask, one with a backward mask) are applied to the visit embeddings, capturing contextual and temporal relationships.
  4. Visit-level Attention: Attention pooling layers are used to aggregate the visit-level representations into a single patient journey representation.
  5. Prediction: Finally, a feed-forward layer with a softmax function generates a categorical distribution for predicting targeted outcomes.

Datasets

The paper utilizes two sub-datasets derived from the MIMIC-III dataset:

  • Dx: This dataset includes only diagnosis codes for each visit.
  • Dx&Tx: This dataset includes both diagnosis and procedure codes for each visit.

The authors pre-processed the data by selecting patients with at least two visits and removing infrequent diagnosis codes (threshold empirically set to 5). This resulted in approximately 7,499 patients with an average of 2.66 visits per patient.

Key Insights and Ideas

  • Temporal Gaps Matter: The paper strongly emphasizes that the time intervals between patient visits are critical and have been largely overlooked by previous methods. MusaNet's interval encoding directly addresses this.
  • Self-Attention for Temporal Data: While self-attention is powerful for capturing contextual relationships, it needs modifications (like mSA) to effectively handle the sequential and temporal nature of patient journeys.
  • Hierarchical Representation: The multi-level structure of EHRs (codes -> visits -> journey) is a key insight. MusaNet leverages this hierarchy through different attention layers.
  • Explainability through Attention: The attention mechanism inherently provides a degree of explainability by highlighting which visits or diagnoses are most important for a prediction.

Key Results

MusaNet consistently outperformed state-of-the-art baseline models on two key healthcare prediction tasks:

  • Future Re-admission (PR-AUC): MusaNet achieved the highest PR-AUC scores on both the Dx and Dx&Tx datasets. For example, on the Dx dataset, MusaNet achieved a PR-AUC of 0.3261, outperforming baselines like RNN (0.3021) and TransEnc (0.3116).
  • Future Diagnosis (Precision@k): MusaNet also demonstrated superior performance in predicting diagnosis categories, achieving higher Precision@k scores across various k values (5, 10, 20, 30) on both datasets. For instance, on Dx&Tx, MusaNet achieved Precision@30 of 0.8242.

The paper also showed that MusaNet is robust to longer sequences of visits, with its performance advantage increasing as the visit sequence length grows.

Limitations and Future Work

  • Data Imbalance: While not explicitly detailed as a limitation, the paper uses PR-AUC, a metric suitable for imbalanced data, suggesting that imbalanced datasets are a common challenge in this domain.
  • External Knowledge: The paper notes that models like GRAM and KAME use external knowledge, which MusaNet does not, implying a potential avenue for future improvement.
  • Generalizability: While tested on MIMIC-III, further validation on diverse EHR datasets would be beneficial.
  • Future Work: The authors suggest that their framework could be extended to other healthcare applications beyond prediction, such as patient similarity or cohort discovery. The explicit modeling of temporal dependencies and hierarchical structure opens doors for more sophisticated patient journey analysis.

Summary updated

Abstract

Understanding patients’ journeys in healthcare system is a fundamental prepositive task for a broad range of AI-based healthcare applications. This task aims to learn an informative representation that can comprehensively encode hidden dependencies among medical events and its inner entities, and then the use of encoding outputs can greatly benefit the downstream application-driven tasks. A patient journey is a sequence of electronic health records (EHRs) over time that is organized at multiple levels: patient, visits and medical codes. The key challenge of patient journey understanding is to design an effective encoding mechanism which can properly tackle the aforementioned multi-level structured patient journey data with temporal sequential visits and a set of medical codes. This paper proposes a novel self-attention mechanism that can simultaneously capture the contextual and temporal relationships hidden in patient journeys. A multi-level self-attention network (MusaNet) is specifically designed to learn the representations of patient journeys that is used to be a long sequence of activities. We evaluated the efficacy of our method on two medical application tasks with real-world benchmark datasets. The results have demonstrated the proposed MusaNet produces higher-quality representations than state-of-the-art baseline methods. The source code is available in https://github.com/xueping/MusaNet .
BibTeX
@inproceedings{peng2020self,
  author = {Xueping Peng and Guodong Long and Tao Shen and Sen Wang and Jing Jiang},
  title = {{Self-attention Enhanced Patient Journey Understanding in Healthcare System}},
  booktitle = {ECML/PKDD},
  year = {2020},
  doi = {10.1007/978-3-030-67664-3\_43},
}

Loading related content...

Figure