A BERT-based Universal Model for Both Within- and Cross-sentence Clinical Temporal Relation Extraction

Chen Lin, Timothy Miller, Dmitriy Dligach, Steven Bethard, Guergana SavovaNAACL 2019

TL;DR

This paper introduces a novel BERT-based approach for extracting temporal relations (like "before," "after," "during") from clinical text, overcoming the limitations of previous sentence-bound methods. By treating text as a continuous sequence of tokens and leveraging BERT's powerful contextual understanding, they achieve state-of-the-art results on a challenging clinical temporal relation extraction task.

What Problem Does This Paper Address?

Extracting temporal relationships between events and time expressions in clinical notes is crucial for understanding disease progression, treatment effectiveness, and for applications like clinical question answering. Traditional methods often struggle because they are designed to work within individual sentences, making them prone to errors when relations span across sentence boundaries. Furthermore, labeled clinical data for this task is scarce, making it difficult to train robust models from scratch. This paper aims to address these limitations by developing a model that can effectively capture temporal relations regardless of sentence structure and leverage pre-trained language models to mitigate data scarcity.

Methods

The core of this work is the application of BERT (Bidirectional Encoder Representations from Transformers) to clinical temporal relation extraction. Instead of relying on sentence-level analysis, the authors treat the text as a continuous sequence of tokens and use a "window-based processing" mechanism. This means they consider pairs of entities (events or time expressions) within a fixed-size window of tokens (e.g., 60 tokens) to capture both within-sentence and cross-sentence relations.

Key methodological contributions include:

  • Sentence-Agnostic BERT: Fine-tuning BERT on a sequence of tokens, effectively making it agnostic to sentence boundaries.
  • Window-Based Processing: Defining candidate relations based on proximity within a token window, rather than strict sentence containment.
  • Domain-Specific Pre-training: Pre-training BERT on clinical data (MIMIC-III) to create a BERT-MIMIC model, and also leveraging BioBERT (pre-trained on biomedical literature).
  • "Silver" Instance Augmentation: Employing a self-training approach to generate additional labeled data ("silver" instances) from unlabeled clinical text, which helps improve model performance by balancing the positive-to-negative instance ratio.
  • Custom Tokenization: Using non-XML tags to mark entity positions and time classes, which are better handled by BERT's WordPiece tokenizer compared to standard XML tags.

Datasets

  • THYME Corpus: This is the primary dataset used for both training and evaluation. It's a widely used benchmark for clinical temporal relation extraction, annotated with time expressions (TIMEX3), events (EVENT), and temporal relations (TLINK). The paper specifically uses the colon cancer and brain cancer subsets.
  • MIMIC-III: Used for pre-training the BERT-MIMIC model. This dataset contains a large volume of electronic medical records from intensive care units.

Key Insights and Ideas

  • BERT's Power Beyond Sentences: The success of BERT in this task highlights its ability to capture long-range dependencies and contextual information, making it a natural fit for problems that traditionally suffer from sentence boundary limitations.
  • Windowing as a Solution: The simple yet effective window-based approach successfully bridges the gap between within-sentence and cross-sentence relation extraction, demonstrating that a unified model can handle both.
  • The Value of Domain-Specific Pre-training: Pre-training BERT on clinical data (BERT-MIMIC) and biomedical literature (BioBERT) provides a performance boost, suggesting that specialized language representations are beneficial for clinical NLP tasks.
  • "Silver" Data is Key: Augmenting the limited gold-standard data with automatically generated "silver" instances significantly improves performance, particularly by addressing the imbalanced nature of relation extraction tasks.
  • Tagging Matters: The choice of how to represent entities and time expressions (e.g., using non-XML tags) can impact BERT's performance, indicating the importance of careful input representation.

Key Results

The paper reports achieving new state-of-the-art results on the THYME corpus:

  • Within-domain: 0.684 F1 score (a 0.055-point improvement over previous best).
  • Cross-domain: 0.565 F1 score (a 0.018-point improvement).

The BERT-based models, especially BERT-TS (BERT fine-tuned with silver instances), BioBERT(pmc)-TS, and BERT-MIMIC-TS, significantly outperformed previous methods, including bidirectional LSTMs. The window-based approach with a 60-token window generally yielded better results than a 50-token window.

Limitations and Future Work

  • Computational Resources: The authors acknowledge that pre-training BERT from scratch on a larger clinical corpus would require significant computational resources, which were a limitation for this study.
  • Error Analysis: While an error analysis was performed, the authors note that it was difficult to subtype errors precisely due to the lack of gold sentence annotations in the THYME corpus and the need to factor in closure over relations.
  • Future Work: The paper suggests pre-training BERT on even larger clinical corpora to further improve performance. They also mention the potential benefit of re-training BERT with added entity tags in its vocabulary to better contextualize them. Further investigation into the impact of different window sizes and the integration of prior knowledge for short-distance relations are also implied.

Summary updated

BibTeX
@inproceedings{lin2019bert,
  author = {Chen Lin and Timothy Miller and Dmitriy Dligach and Steven Bethard and Guergana Savova},
  title = {{A BERT-based Universal Model for Both Within- and Cross-sentence Clinical Temporal Relation Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure