A BERT-based One-Pass Multi-Task Model for Clinical Temporal Relation Extraction

Chen Lin, Timothy A. Miller, Dmitriy Dligach, Farig Sadeque, Steven Bethard, Guergana SavovaBioNLP 2020 · DOI · OpenReview

Introduction

Extracting timelines from Electronic Medical Records (EMRs) is essential for longitudinal patient analysis, including disease progression and treatment efficacy. This task involves identifying medical events, time expressions, and their temporal relationships (e.g., "Event A contains Event B"). While BERT-based models have set new benchmarks in this field, they typically rely on a computationally expensive multi-pass approach, processing the same sentence for every candidate entity pair. This paper introduces a "one-pass" architecture that extracts all relations in a single forward pass, significantly improving efficiency without compromising accuracy.

Methodology

The proposed model shifts from standard relation classification to a comprehensive relation extraction framework.

One-Pass Encoding and Global Context

Unlike traditional models that re-encode sequences for each pair, this architecture encodes the input sequence once. To handle long-distance relations common in clinical narratives, the model utilizes an entity-aware self-attention mechanism. A critical innovation is the inclusion of a global embedding ($G$), created by average-pooling the entire sequence. This global context is concatenated with local entity embeddings ($e1, e2$) to facilitate broader inference.

Architecture of the one-pass multi-task model featuring global pooling and dual task heads.

Multi-Task Learning (MTL)

The model is trained jointly on two distinct but related tasks:

  1. CONTAINS: Determining the temporal containment relationship between two entities.
  2. DocTimeRel: Classifying the temporal status of an entity relative to the Document Creation Time (e.g., BEFORE, AFTER, OVERLAP).

This multi-task setup serves as a regularizer, enhancing the model's ability to generalize across different medical domains.

Experimental Setup

The model was evaluated using the THYME corpus, which contains annotated clinical text. The Colon Cancer subset was used for primary training and development, while the Brain Cancer subset served as a holdout set to test cross-domain generalizability.

Results and Discussion

The one-pass model demonstrates that "Green AI" can achieve state-of-the-art results while being environmentally and computationally responsible.

Efficiency Gains

The one-pass approach reduces the number of training instances from approximately 497,000 to just 20,000. This results in a 25-fold increase in efficiency compared to multi-pass baselines, drastically reducing GPU compute requirements.

Efficiency comparison showing the reduction in flops and training instances.

Performance and Generalization

On the Colon Cancer test set, the model achieved an F1 score of 0.686 on the CONTAINS task, matching the performance of multi-pass models that required additional silver data. Furthermore, on the unseen Brain Cancer domain, the MTL model outperformed previous SOTA models with an F1 of 0.582, proving its robustness.

Conclusion

This research proves that architectural optimizations can eliminate the need for redundant computation in clinical NLP. By combining one-pass encoding with global context and multi-task learning, the authors provide a lightweight yet powerful framework suitable for deployment with various domain-specific pre-trained language models.

Summary updated

Abstract

Recently BERT has achieved a state-of-the-art performance in temporal relation extraction from clinical Electronic Medical Records text. However, the current approach is inefficient as it requires multiple passes through each input sequence. We extend a recently-proposed one-pass model for relation classification to a one-pass model for relation extraction. We augment this framework by introducing global embeddings to help with long-distance relation inference, and by multi-task learning to increase model performance and generalizability. Our proposed model produces results on par with the state-of-the-art in temporal relation extraction on the THYME corpus and is much “greener” in computational cost.
BibTeX
@inproceedings{lin2020bert,
  author = {Chen Lin and Timothy A. Miller and Dmitriy Dligach and Farig Sadeque and Steven Bethard and Guergana Savova},
  title = {{A BERT-based One-Pass Multi-Task Model for Clinical Temporal Relation Extraction}},
  booktitle = {BioNLP},
  year = {2020},
  doi = {10.18653/v1/2020.bionlp-1.7},
}
Figure