Ensemble-based Fine-Tuning Strategy for Temporal Relation Extraction from the Clinical Narrative

Lijing Wang, Timothy M. Miller, Steven Bethard, Guergana Savova2022 · DOI

TL;DR

This paper explores how to get more mileage out of powerful pre-trained language models for clinical text analysis by using ensemble methods during the fine-tuning process. They found that combining multiple fine-tuned models, especially using a dynamic snapshot approach, significantly boosts performance on temporal relation extraction from clinical narratives, outperforming single, hyperparameter-optimized models.

What Problem Does This Paper Address?

Electronic Medical Records (EMRs) are treasure troves of information that can improve patient care, but extracting this information is challenging. Clinical Natural Language Processing (NLP) aims to unlock this data, and recent advances in transformer-based models like BERT have shown great promise. However, clinical data is often scarce and sensitive, making it difficult to train large models from scratch. The common approach is to fine-tune pre-trained models on clinical data. This fine-tuning process involves optimizing many hyperparameters, which is computationally expensive and often leads to models that are good but not necessarily the best. The authors ask: can we leverage the effort already put into hyperparameter optimization and further improve performance by using ensemble methods during fine-tuning, without excessive additional computational cost?

Methods

The core idea is to use ensemble learning to combine the predictions of multiple fine-tuned models. They start with a strong baseline model (PubmedBERTbase-MimicBig-EntityBERT) and its fine-tuning setup. Then, they explore two main ensemble strategies:

  • Bagging Ensemble: This involves training multiple models independently and combining their predictions, often using majority voting. They investigate diversity introduced by varying:
  • Random Seeds: Different initializations for training.
  • Learning Rate Schedulers: Different ways to adjust the learning rate during training (linear, cosine with restarts, polynomial).
  • Epoch Numbers: Training for different numbers of passes over the data.
  • Pretrained Models: Using different variants of PubMedBERT.
  • Data Shuffling: Different ways of mixing training and validation data.
  • Dynamic Snapshot Ensemble (DynSnap-ENS): This is a more sophisticated approach where a model is fine-tuned sequentially. During each sequential training run, the model periodically saves its best checkpoints (snapshots) based on validation performance. A dynamic pruning algorithm then selects a subset of these snapshots to form the final ensemble. This method aims to capture diverse model states from a single, computationally efficient framework. They specifically use a cyclic learning rate schedule and save multiple snapshots per training run.

Datasets

The primary dataset used is THYME++, a corpus of de-identified clinical and pathology notes related to colon and brain cancer patients. This dataset is annotated for temporal relations and has been used in previous SemEval shared tasks. They use the train/dev/test splits as described by Wright-Bettner et al. (2020).

Key Insights and Ideas

  • Ensembling is a powerful fine-tuning strategy: Even after optimizing hyperparameters for a single model, combining multiple fine-tuned models can yield significant performance gains.
  • Dynamic Snapshot Ensembling shines: DynSnap-ENS, which intelligently selects diverse model checkpoints from sequential training runs, proved to be the most effective ensemble method.
  • Diversity matters, but not all sources are equal: While varying random seeds and data shuffling provided good improvements, diversity in the number of training epochs was particularly impactful. Diversity in learning rate schedulers and pretrained model variants yielded smaller gains.
  • Computational efficiency is key: DynSnap-ENS offers a way to explore multiple sources of diversity (data, model parameters, epochs, learning rate) within a single framework, making it more computationally efficient than independent hyperparameter searches.

Key Results

The ensemble methods consistently improved performance over the baseline model (BASE). Specifically:

  • DynSnap-ENS achieved the highest overall F1 score improvement, a 2.8% absolute increase. This was primarily driven by a 6.2% absolute increase in precision.
  • Among the bagging ensembles, Epoch-ENS (diversity in epoch number) showed the largest improvement, 2.4% absolute F1.
  • Data shuffling and random seeds also provided notable improvements (1.7% and 1.3% F1, respectively).

Limitations and Future Work

The authors acknowledge a few limitations and suggest future directions:

  • Reproducibility Challenges: They note that they were unable to perfectly reproduce the exact scores reported in a previous study (Lin et al., 2021) due to differences in implementation scripts and missing settings.
  • Ensemble Size: The paper explores a fixed ensemble size (5 for bagging, and dynamically determined for DynSnap-ENS). Future work could investigate the impact of varying ensemble sizes on performance.
  • Snapshot Strategy and Pruning: Further exploration of the snapshot strategy and pruning algorithm within DynSnap-ENS could lead to even better results.
  • Broader Task Applicability: The authors suggest testing the trained ensemble learners on a wider range of clinical domain tasks beyond temporal relation extraction.

Summary updated

Abstract

In this paper, we investigate ensemble methods for fine-tuning transformer-based pretrained models for clinical natural language processing tasks, specifically temporal relation extraction from the clinical narrative. Our experimental results on the THYME data show that ensembling as a fine-tuning strategy can further boost model performance over single learners optimized for hyperparameters. Dynamic snapshot ensembling is particularly beneficial as it finetunes a wide array of parameters and results in a 2.8% absolute improvement in F1 over the base single learner.
BibTeX
@misc{wang2022ensemble,
  author = {Lijing Wang and Timothy M. Miller and Steven Bethard and Guergana Savova},
  title = {{Ensemble-based Fine-Tuning Strategy for Temporal Relation Extraction from the Clinical Narrative}},
  year = {2022},
  doi = {10.18653/v1/2022.clinicalnlp-1.11},
}
Figure