Bidirectional RNN for Medical Event Detection in Electronic Health Records

Abhyuday N Jagannatha, Hong YuNAACL 2016

Sure, here's a blog-style summary of the paper "Bidirectional RNN for Medical Event Detection in Electronic Health Records":

TL;DR

This paper introduces the use of Bidirectional Recurrent Neural Networks (RNNs), specifically LSTMs and GRUs, for extracting medical events and their attributes from noisy Electronic Health Records (EHRs). The study demonstrates that these RNN models significantly outperform traditional Conditional Random Field (CRF) models, particularly in improving recall for various medical event types.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) are a rich source of patient information, but they are often unstructured and noisy, containing incomplete sentences, abbreviations, and medical jargon. Extracting meaningful medical events (like diagnoses, medications, and adverse drug events) from this text is crucial for various clinical applications, such as drug surveillance and understanding patient health trajectories. Traditional methods, like CRFs, rely on fixed context windows, which struggle to capture the long-range dependencies and varying contextual needs of different medical event types. For instance, identifying a medication might only require a few words, while an adverse drug event might be described across an entire sentence. This paper aims to address this limitation by leveraging RNNs, which are better equipped to handle sequential data and learn long-term dependencies.

Methods

The authors explore two types of RNNs: Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). Both are designed to overcome the vanishing gradient problem in traditional RNNs, enabling them to learn dependencies over longer sequences.

  • LSTM: Uses a complex gating mechanism (input, forget, and output gates) and a cell state to control information flow and maintain long-term memory.
  • GRU: A simpler variant of LSTM, using an update gate and a reset gate to manage information flow.

The paper employs a bidirectional architecture for both LSTM and GRU. This means the network processes the input sequence in both forward and backward directions, concatenating the outputs to create a richer representation of the word and its context. This is crucial for capturing dependencies from both preceding and succeeding words.

The models are trained using skip-gram word embeddings initialized from a large unlabeled biomedical corpus. This helps the models handle words not frequently seen in the labeled data. For comparison, they also implement CRF models, including a baseline CRF with no context features and a CRF with added context features (bag-of-words representation of surrounding text) to mimic some of the contextual information captured by RNNs.

Datasets

The primary dataset used is a corpus of 780 English EHR notes from cancer patients with hematological malignancy, totaling 613,593 word tokens. This corpus was annotated by at least two annotators with a high inter-annotator agreement (0.93 kappa). The annotated events include:

  • Medication: Drugname, Dosage, Frequency, Duration, Route
  • Disease: ADE (Adverse Drug Event), Indication, Other SSD (Sign, Symptom, Disease)

The paper also mentions using a large unlabeled biomedical corpus for training word embeddings, comprising:

  • English Wikipedia (2015)
  • An unlabeled EHR corpus (99,700 notes)
  • PubMed Open Access articles

Key Insights and Ideas

  • RNNs excel at capturing long-range dependencies: The core insight is that medical event detection in EHRs benefits significantly from models that can learn dependencies across variable context lengths, a strength of RNNs over traditional fixed-window methods.
  • Bidirectionality is key: Processing text in both forward and backward directions allows the models to leverage context from both sides of a word, leading to more accurate predictions.
  • GRU slightly outperforms LSTM: In this specific study, the GRU model, particularly the document-level GRU, showed a slight edge over LSTM in overall F-score, possibly due to its simpler architecture being more efficient on the given dataset size.
  • Context is crucial for rare events: The paper highlights that rare events like Adverse Drug Events (ADEs) and indications are particularly challenging and benefit most from contextual information.
  • "Outside" confusion is a major error source for CRFs: A significant portion of CRF errors stems from misclassifying medical terms as "Outside" (false negatives) or vice-versa (false positives). RNNs significantly reduce this confusion.

Key Results

  • Significant Outperformance: RNN models (LSTM and GRU) significantly outperformed the baseline CRF-context model.
  • GRU-document leads: The GRU-document model achieved the highest F-score (0.8031), improving recall, precision, and F-score by 19%, 2%, and 11% respectively compared to the CRF-context baseline.
  • Label-wise gains: GRU-document showed the largest gains in six out of nine tags, with a notable improvement for "indication" (nearly 50% increase).
  • Sentence vs. Document models: Sentence-level models performed better than document-level models for ADE and Dosage, suggesting that local context is sufficient for these specific event types.
  • Robustness with less data: Figure 4 indicates that RNN models maintain strong performance even with reduced training data sizes, outperforming CRFs in such scenarios.

Limitations and Future Work

  • Data Sparsity for Rare Labels: The paper notes that the BILOU tagging scheme was not used due to data sparsity for rarer labels.
  • No Explicit Output Constraints: The authors acknowledge that RNNs do not explicitly enforce constraints or dependencies over the output labels. They suggest that combining RNNs with a probabilistic graphical model framework for structured output prediction could further improve performance.
  • Dataset Size: While the unlabeled corpus is large, the labeled EHR corpus is relatively small (780 documents). The authors speculate that with a larger dataset, LSTM might perform comparably to or better than GRU.
  • Future Work: The primary future direction suggested is integrating RNNs with probabilistic graphical models to enforce output constraints.

Summary updated

BibTeX
@inproceedings{jagannatha2016bidirectional,
  author = {Abhyuday N Jagannatha and Hong Yu},
  title = {{Bidirectional RNN for Medical Event Detection in Electronic Health Records}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
}
Figure