UTSA-NLP at ChemoTimelines 2024: Evaluating Instruction-Tuned Language Models for Temporal Relation Extraction

Xingmeng Zhao, Anthony RiosNAACL 2024

TL;DR

This paper explores using instruction-tuned Large Language Models (LLMs) for temporal relation extraction in clinical notes, specifically for chemotherapy timelines. They found that while incorporating document context improves performance, further pre-training on unlabeled data didn't yield significant gains, possibly due to limited data usage. The study highlights the ongoing challenge of adapting general LLMs to the specialized and rapidly evolving medical domain, where they still underperform traditional domain-specific models.

What Problem Does This Paper Address?

Extracting chemotherapy treatment timelines from Electronic Health Records (EHRs) is crucial for advancing cancer research and improving patient care. This involves identifying chemotherapy-related events and their temporal relationships. While domain-specific models like BioBERT and ClinicalBERT have been successful, they struggle with adaptability to new or evolving medical terminology and contexts, leading to performance drops. Large Language Models (LLMs) show promise due to their few-shot and zero-shot learning capabilities, but adapting them to the medical domain, with its unique jargon and nuances, remains a challenge. This paper addresses the gap in effectively leveraging instruction-tuned LLMs for this specific clinical NLP task, aiming to create an end-to-end system that can handle the complexities of medical text.

Methods

The authors propose a three-step instruction-tuning strategy for open-source LLMs (specifically OpenChat-3.5-7B) to tackle temporal relation extraction in clinical notes.

  1. Instruction-tuning on QA pairs: The LLM is first fine-tuned on question-answer pairs designed for Named Entity Recognition (NER) and Relation Extraction (RE). For RE, a three-step reasoning framework is employed, combining concept-level entity knowledge and explicit textual evidence to generate answers. For NER, a Chain-of-Thought prompting approach is used to guide the model in identifying entities.
  2. Joint instruction-tuning on QA pairs and documents: This step enhances the model's ability to absorb knowledge from complex clinical documents by training on both QA pairs and their associated full documents. The documents are strategically placed after the QA pairs to improve knowledge assimilation.
  3. Integrating Unlabeled Corpus: The final step involves continued pre-training on a large, unlabeled clinical corpus. The intuition here is to help the model learn specialized medical vocabulary and context without the risk of catastrophic forgetting, by building upon the knowledge gained from the previous two steps.

Datasets

The study utilizes both unlabeled and labeled EHR data from the University of Pittsburgh/UPMC for the ChemoTimelines 2024 shared task.

  • Unlabeled Data: Approximately 62,000 patients with breast and ovarian cancer, and 16,000 patients with melanoma.
  • Labeled Data: Gold annotations for 310 patients, focusing on EVENTS (chemotherapy treatments), TIMEX3s (temporal expressions), and temporal relations (TLINKs) between them. The training set includes data from ovarian, breast, and melanoma patients, with a similar split for the development set. All data was de-identified.

Key Insights and Ideas

  • Document context is key: Incorporating full clinical documents alongside QA pairs during instruction tuning significantly improves performance, suggesting LLMs benefit from richer contextual information beyond just question-answer formats.
  • Unlabeled data's limited impact: While continuous pre-training on unlabeled data is a promising avenue for domain adaptation, its effectiveness was limited in this study, possibly due to using only a small fraction (1%) of the available data. This suggests that the quantity and strategy of unlabeled data usage are critical.
  • LLMs vs. Domain-Specific Models: Despite the advancements in LLMs, they still struggle to match the performance of established domain-specific models like EntityBERT for tasks requiring high specificity in the medical domain. This highlights the trade-off between LLMs' generalizability and the specialized knowledge of fine-tuned models.
  • Challenges in Negative Sampling: Generating accurate negative examples (i.e., identifying when no relation exists) for training is a significant challenge, leading to false positive relation triples and impacting model performance.

Key Results

  • Subtask 1 (Relation Extraction): Instruction tuning with QA pairs and associated documents achieved the best performance on the development set, with an overall average score of .68. On the test set, their method ranked mid-tier with an average precision of .69.
  • Subtask 2 (Entity and Relation Extraction): Performance was more variable across disease types, with a total average precision of .47 on the development set and a significantly lower .22 on the test set, trailing behind the baseline.
  • Impact of Unlabeled Data: Adding unlabeled data resulted in a slight decrease in performance on the development set for subtask 1, likely due to limited usage.

Limitations and Future Work

  • Limited Unlabeled Data: The study was constrained by time and resources, leading to the use of less than 1% of the unlabeled data, which likely hindered the full exploitation of continuous training capabilities.
  • Sentence-Level Focus: The experiments primarily considered entities within the same sentence, overlooking cases where entities might span multiple sentences, which could impact performance.
  • Prompt Sensitivity: The effectiveness of NER and RE tasks is known to be sensitive to prompt design, and the initial single prompt strategy might not have been optimal.
  • Future Work: The authors suggest exploring outcome supervision and process-based reward mechanisms in reinforcement learning to address false positive predictions. Further research is also needed to evaluate more comprehensive training strategies and ablation studies to optimize performance.

Summary updated

BibTeX
@inproceedings{zhao2024utsa,
  author = {Xingmeng Zhao and Anthony Rios},
  title = {{UTSA-NLP at ChemoTimelines 2024: Evaluating Instruction-Tuned Language Models for Temporal Relation Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure