Patient Risk Assessment and Warning Symptom Detection Using Deep Attention-Based Neural Networks

Ivan Girardi, Pengfei Ji, An-phi Nguyen, Nora Hollenstein, Adam Ivankay, Lorenz Kuhn, Chiara Marchiori, Ce ZhangEMNLP 2018

TL;DR

This paper introduces an attention-based convolutional neural network (ACNN) for automatically assessing patient risk and detecting critical "warning symptoms" from medical notes. By combining deep learning with an attention mechanism, the model not only classifies urgency but also provides explainable insights into why a recommendation is made, a crucial step towards transparent AI in healthcare.

What Problem Does This Paper Address?

In healthcare, efficient and accurate patient triage is paramount. Intelligent triage systems aim to guide patients to the appropriate level of care and urgency. However, existing systems often lack transparency, making it difficult for users to understand the rationale behind their recommendations. Furthermore, accurately identifying "warning symptoms" – those that signal a potential emergency – is critical for distinguishing urgent cases from less severe ones. This paper tackles the challenge of building a robust, explainable triage system that can both assess overall patient risk and pinpoint specific warning symptoms from unstructured medical text.

Methods

The core of this work is an Attention-based Convolutional Neural Network (ACNN). Here's a breakdown of their approach:

  • N-Gram Encoder: They use a 2D convolutional layer to process sequences of words (n-grams) from medical notes. This captures local contextual information, allowing the model to identify patterns that might be indicative of specific symptoms or conditions. A key advantage highlighted is its ability to detect symptom combinations that might be harmless individually but dangerous together (e.g., pain in arm + sudden nausea).
  • N-Gram Level Attention Layer: This is where the "attention" comes in. For each n-gram, the model learns an "attention weight" that signifies its importance to the overall document's meaning. This allows the model to focus on the most relevant parts of the text.
  • Warning Symptom Detection: The attention scores are leveraged to identify warning symptoms. By analyzing which symptoms receive high attention weights, especially in the context of urgent care cases, the model can pinpoint critical indicators. They developed a method to quantify this by calculating an "attention score" for each symptom.
  • Explainability: The attention mechanism inherently provides a degree of explainability. By visualizing the attention scores, users can see which symptoms contributed most to the model's prediction, offering a rationale for the recommended triage level.

Datasets

The researchers utilized a substantial dataset of 600,000 case records written in German collected over five years. This dataset was curated by selecting cases treated by top-ranked doctors. For evaluation, they also created a "symptoms dataset" containing only selected medical entities extracted from the full text.

Key Insights and Ideas

  • Symptom Combinations Matter: The ACNN's ability to process n-grams and use attention allows it to identify dangerous symptom combinations that might be missed if symptoms were analyzed in isolation. This is a crucial insight for medical diagnosis.
  • Attention as Explainability: The attention mechanism isn't just for improving performance; it's a direct pathway to explainability. Visualizing attention scores allows users to understand why a patient is classified as urgent, building trust in the AI system.
  • Full Text vs. Extracted Entities: The paper benchmarks performance using both the full medical text and a reduced set of extracted medical entities. While both approaches yield reasonable results, the full text generally performs better, underscoring the importance of contextual information beyond just symptom names.
  • Confidence Thresholding Boosts Precision: Applying a confidence threshold (e.g., 0.6) to the model's predictions significantly increases precision, albeit by discarding a portion of the cases. This is a practical strategy for deploying such systems in real-world scenarios where high precision is critical.

Key Results

  • High Precision: The ACNN achieved a precision of 79% on the full-text dataset and 66% on the symptoms dataset.
  • Improved Precision with Confidence: When a confidence threshold of 0.6 was applied, precision increased to 85% (full text) and 75% (symptoms dataset).
  • Competitive Performance: The ACNN demonstrated competitive performance compared to other deep learning models like Kim CNN and CLSTM, particularly in the urgent care and telecare classes.
  • Warning Symptom Detection Effectiveness: Dropping attention-detected warning symptoms led to the largest performance decrease, validating their informativeness.

Limitations and Future Work

  • Dataset Specificity: The model was trained on German medical notes, which may include Swiss German expressions. Adapting it to other languages would require retraining.
  • Data Imbalance: While efforts were made to mitigate class imbalance, the original distribution of ten recommendation classes was uneven.
  • Future Work: The authors suggest exploring additional techniques, such as hierarchical decision trees, to further minimize medical risk. They also aim to apply the model in an industrial triage application.

Summary updated

BibTeX
@inproceedings{girardi2018patient,
  author = {Ivan Girardi and Pengfei Ji and An-phi Nguyen and Nora Hollenstein and Adam Ivankay and Lorenz Kuhn and Chiara Marchiori and Ce Zhang},
  title = {{Patient Risk Assessment and Warning Symptom Detection Using Deep Attention-Based Neural Networks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure