Attention Neural Model for Temporal Relation Extraction

Sijia Liu, Liwei Wang, Vipin Chaudhary, Hongfang Liu2019 · DOI

TL;DR

This paper introduces an attention-based GRU neural network model for extracting temporal "containment" relationships between events and time expressions in clinical text. The model achieves state-of-the-art performance on the THYME corpus, demonstrating the effectiveness of attention mechanisms in pinpointing crucial information for this task.

What Problem Does This Paper Address?

Extracting structured information from unstructured clinical text, like electronic health records (EHRs), is crucial for various healthcare applications, including clinical decision support and predictive modeling. A key challenge within this domain is temporal relation extraction, specifically identifying how events and time expressions relate to each other. This paper focuses on "containment" relations, where an event occurs within a specific time frame (e.g., "patient was monitored for three months"). Existing methods, while improving, still face challenges in accurately capturing these nuanced temporal links, especially within the complex language of clinical narratives.

Methods

The authors propose an attention-based neural network model built upon a Gated Recurrent Unit (GRU) architecture. Here's a breakdown of their approach:

  • Input Representation: Clinical text is first tokenized into sentences. Entities (events and time expressions) are then marked with special XML tags (e.g., <e>...</e> for events, <t>...</t> for time). This tagged sequence serves as the input to the model. They found this XML tag representation to be more effective than simple positional embeddings.
  • Instance Encoding: For each potential relation between two entities within a sentence, the model encodes the relevant sequence. This involves:
  • Word Embeddings: Pre-trained 300-dimensional GloVe embeddings are used, with special handling for out-of-vocabulary words and XML tags.
  • RNN Layer (GRU): A GRU processes the sequence of word embeddings to capture temporal dependencies.
  • Attention Mechanism: This is the core innovation. An attention layer learns to assign weights to different parts of the RNN's output, effectively focusing on the most relevant words or tokens for determining the temporal relation. A trainable context vector is used to guide this attention. The attention weights are then used to compute a weighted sum of the RNN outputs, creating a sentence vector that highlights salient information.
  • Classification: The resulting sentence vector is fed into a fully connected layer, followed by a softmax function to predict one of three labels: "CONTAINS," "CONTAINED," or "NA" (no containment relation).

Datasets

The primary dataset used is the THYME (Temporal Histories of Your Medical Event) corpus. This corpus is derived from clinical notes of 200 patients and contains manually annotated events and time expressions (Timex3). The authors focus on intra-sentence temporal relations, meaning they consider relations only between entities within the same sentence.

Key Insights and Ideas

  • Attention is Key for Clinical Text: The paper strongly suggests that attention mechanisms are highly effective for temporal relation extraction in clinical narratives. By allowing the model to dynamically focus on salient parts of the input, it can better identify the subtle cues that indicate temporal relationships.
  • GRU Outperforms LSTM (in this context): While both GRU and LSTM are powerful RNNs, the authors found GRU to perform slightly better. They hypothesize this might be due to GRU having fewer trainable parameters, leading to better convergence on datasets with a limited number of positive relational instances.
  • XML Tagging Improves Representation: Using XML tags to explicitly mark entities within the input sequence proved more beneficial than relying solely on positional information. This provides a clearer signal to the model about the nature of the tokens.
  • Intra-sentence Focus is Sufficient for High Performance: By focusing solely on intra-sentence relations, the model achieved state-of-the-art results, suggesting that a significant portion of temporal information can be captured at this granular level.

Key Results

The proposed attention-based GRU (ATT-GRU) model achieved state-of-the-art performance on the THYME corpus for intra-sentence temporal relation extraction.

  • Event-Time Relations: ATT-GRU achieved an F1 score of 0.750, outperforming the CNN model by 0.050.
  • Event-Event Relations: ATT-GRU achieved an F1 score of 0.579, comparable to other neural models and slightly better than the CNN model, though not as good as a feature-based SVM.
  • Combined Relations (Table 2): When considering all intra-sentence relations, the ATT-GRU model achieved an F1 score of 0.690, which was favorably comparable to other neural models like BiLSTM incorporating cTAKES outputs.

Limitations and Future Work

  • Limitations:
    • The authors acknowledge that their model, while strong, is not as good as the feature-based SVM for Event-Event relations on the THYME system.
  • The study focused solely on intra-sentence relations, excluding inter-sentence relationships.
  • The model does not explicitly handle class imbalance as effectively as some traditional methods (e.g., by tuning class weights), which can lead to confusion between "NA" and positive relation instances.
  • Future Work:
    • Hierarchical Models: Incorporating character-level embeddings into the word-level representation.
  • Advanced Attention Mechanisms: Exploring multi-head attention and self-attention mechanisms.
  • Inter-sentence Relations: Extending the model to capture temporal relations across sentences.
  • Leveraging Entity Attributes: Utilizing finer-grained entity attributes (like UMLS semantic types) to further improve performance.

Summary updated

Abstract

Neural network models have shown promise in the temporal relation extraction task. In this paper, we present the attention based neural network model to extract the containment relations within sentences from clinical narratives. The attention mechanism used on top of GRU model outperforms the existing state-of-the-art neural network models on THYME corpus in intra-sentence temporal relation extraction.
BibTeX
@misc{liu2019attention,
  author = {Sijia Liu and Liwei Wang and Vipin Chaudhary and Hongfang Liu},
  title = {{Attention Neural Model for Temporal Relation Extraction}},
  year = {2019},
  doi = {10.18653/v1/w19-1917},
}
Figure