LAILab at Chemotimelines 2024: Finetuning sequence-to-sequence language models for temporal relation extraction towards cancer patient undergoing chemotherapy treatment

Shohreh Haddadan, Tuan-Dung Le, Thanh Duong, Thanh ThieuNAACL 2024

TL;DR

This paper tackles the challenge of automatically extracting chemotherapy timelines from electronic health records (EHRs) using large language models (LLMs). By finetuning instruction-tuned LLMs like Flan-T5, the researchers achieved state-of-the-art results in both identifying temporal relations between medical events and time expressions, and in directly generating patient-specific chemotherapy timelines.

What Problem Does This Paper Address?

Electronic health records (EHRs) are a treasure trove of information for understanding patient journeys, especially for cancer patients undergoing complex chemotherapy treatments. However, the sheer volume and unstructured nature of these records make manual extraction of treatment timelines incredibly time-consuming and impractical. Automatically extracting these timelines is crucial for evaluating treatment efficacy, summarizing patient histories for physicians, and advancing medical research. This paper focuses on two key subtasks within this problem: 1) identifying and classifying temporal relationships between chemotherapy events and time expressions, and 2) directly constructing complete patient chemotherapy timelines from EHR notes.

Methods

The core of this work lies in leveraging the power of Large Language Models (LLMs). For the first subtask (temporal relation classification), the authors reformulated the problem as a text generation task. They finetuned instruction-tuned LLMs, specifically Flan-T5, by providing it with instructions and examples of temporal relations. They also explored different context-gathering strategies (bounded vs. concatenated sentences) to improve performance.

For the second subtask (end-to-end timeline extraction), two main approaches were investigated:

  1. Sequence-to-Sequence Finetuning: They trained a sequence-to-sequence model to directly generate linearized triplets of <event, relation_type, time_expression>. This involved adapting a triplet linearization algorithm and adding custom tokens for relation types to the tokenizer.
  2. Pipeline Approach: This method combined a rule-based system for extracting events and time expressions with their best-performing model from the first subtask for relation classification.

A key technique employed across their experiments is Parameter-Efficient Finetuning (PEFT), specifically LoRA (Low-Rank Adaptation). This allows for efficient finetuning of massive LLMs by freezing most of the pre-trained weights and only training a small number of additional parameters, significantly reducing computational costs.

Datasets

The research utilized a dataset provided for the Chemotimelines shared task, which includes annotated chemotherapy events and time expressions from EHR notes across three cancer types: breast, melanoma, and ovarian. This dataset was used for both training and evaluation. The paper does not mention the creation of a new benchmark dataset.

Key Insights and Ideas

  • Instruction Tuning is Powerful: Finetuning general-domain LLMs with specific instructions (like Flan-T5) proved highly effective for domain-specific tasks like temporal relation extraction, even on unseen data.
  • Context Matters (Sometimes): For temporal relation classification, using "bounded context" (including sentences between the event and time expression) showed a marginal improvement over "concatenated context," particularly for identifying non-relations.
  • LLMs Excel at Generation: The sequence-to-sequence finetuning approach for end-to-end timeline extraction demonstrated strong performance, highlighting the generative capabilities of LLMs for complex structured output.
  • PEFT is Essential: LoRA enabled efficient finetuning of large models, making this research feasible within computational constraints and demonstrating comparable or superior performance to full finetuning.
  • Model Size Correlates with Performance: Larger LLMs generally yielded better results in the second subtask, suggesting that more parameters can capture more nuanced temporal information.

Key Results

The proposed methods, particularly the end-to-end approach using Flan-T5-xxl with LoRA, achieved the highest average scores on the test results as announced by the organizers for both subtasks.

  • Subtask 1 (Temporal Relation Classification): Finetuned Flan-T5-xxl with instruction and bounded context achieved the highest scores across most metrics on the development set.
  • Subtask 2 (Timeline Extraction): The end-to-end approach with Flan-T5-xxl + LoRA achieved the highest precision rates and outperformed the baseline system for melanoma and ovarian cancer on the development set. On the test set, this approach also secured the highest average score among all submissions.

Limitations and Future Work

The authors acknowledge several limitations:

  • Computational Resources: Experiments were constrained by available computational power, limiting the exploration of Flan-T5 without LoRA and preventing testing on other datasets.
  • Data Scarcity: Annotated medical data for such specific tasks is scarce, limiting the ability to generalize findings to different datasets.
  • Legal/Data Agreements: The use and release of the trained models are legally bound by data usage agreements.
  • Explainability: As with most deep learning models, the lack of explainability and interpretability is a limitation.

Future work directions include:

  • Augmenting data for low-frequency relation types.
  • Harnessing unlabeled data for pre-training LLMs.
  • Investigating the impact of these approaches on cancer patient EHR notes.
  • Further exploring the impact of different context strategies and relation types.

Summary updated

BibTeX
@inproceedings{haddadan2024lailab,
  author = {Shohreh Haddadan and Tuan-Dung Le and Thanh Duong and Thanh Thieu},
  title = {{LAILab at Chemotimelines 2024: Finetuning sequence-to-sequence language models for temporal relation extraction towards cancer patient undergoing chemotherapy treatment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure