TL;DR
This paper introduces the 2024 Shared Task on Chemotherapy Treatment Timeline Extraction, a challenge focused on automatically constructing patient-level timelines of chemotherapy treatments from Electronic Health Records (EHRs). The task highlights the difficulty of extracting and ordering complex medical events, with fine-tuned smaller language models outperforming larger models prompted for zero-shot learning, suggesting that sophisticated prompting or larger models are needed for optimal performance.
What Problem Does This Paper Address?
Extracting comprehensive timelines of medical treatments, particularly complex chemotherapy regimens, from unstructured clinical notes in Electronic Health Records (EHRs) is a significant challenge. While medication records might list drugs, they often lack the context of specific protocols, timing, and reasons for changes. Clinical notes can contain this crucial information, but it's embedded within free text, making automated extraction difficult. Previous work has focused on specific aspects like temporal relation extraction, but this shared task aims to create complete patient-level timelines by integrating event extraction, time expression identification, and temporal relation classification, mimicking a real-world clinical research scenario.
Methods
The shared task was divided into two subtasks:
- Subtask 1: Participants were given gold-standard chemotherapy event mentions and time expressions, along with the EHR notes, to construct patient-level timelines. This subtask focused on temporal relation extraction and timeline construction given perfect input.
- Subtask 2: This subtask simulated a more realistic scenario where only EHR notes were provided, requiring an end-to-end system for chemotherapy treatment timeline extraction.
The organizers provided a baseline system using the cTAKES NLP pipeline for preprocessing, Huggingface Transformers for model training, and CLUlab Timenorm for time expression normalization. Participants employed a wide range of methods, including Logistic Regression, TF-IDF, n-grams, transformer models, zero-shot prompting with Large Language Models (LLMs), and instruction tuning. Notably, fine-tuning smaller LMs often yielded superior results compared to prompting larger LLMs.
Datasets
The shared task utilized a dataset derived from the University of Pittsburgh Medical Center (UPMC) EHRs. This dataset included:
- Labeled Dataset: A subset of EHRs with gold annotations for chemotherapy events, time expressions, temporal relations, and patient-level timelines. This dataset was split into training, development, and test sets.
- Unlabeled Dataset: A larger collection of UPMC EHR notes for over 73,000 patients with breast and ovarian cancer (2004-2020) and approximately 15,946 patients with melanoma (2010-2020). This dataset was intended for pre-training or continued training of language models.
The annotation schema followed the THYME2 standard, widely used in clinical temporal relation extraction.
Key Insights and Ideas
- Fine-tuning vs. Prompting: A striking finding is that fine-tuned smaller LMs generally outperformed larger LLMs used with zero-shot prompting. This suggests that for complex, domain-specific tasks like clinical timeline extraction, fine-tuning on relevant data remains a highly effective strategy, and current prompting techniques may not fully leverage the capabilities of large models for such tasks.
- End-to-End is Harder: The significant performance drop between Subtask 1 (with gold inputs) and Subtask 2 (end-to-end extraction) highlights that event and time expression extraction, often considered the "easier" part, is still a challenging problem in real-world clinical text.
- Granularity Matters: The evaluation metric focused on a "relaxed-to-month" granularity for temporal relations, acknowledging that precise day-level accuracy is often less critical in clinical practice than month-level understanding.
Key Results
The shared task saw 18 submissions for Subtask 1 and 9 for Subtask 2. The top-performing teams, such as LAILab and Wonder, primarily utilized fine-tuned language models. For instance, LAILab achieved high scores by fine-tuning Flan-T5-xxl and Bart-large. The performance gap between Subtask 1 and Subtask 2 was substantial, with official scores dropping by at least 0.2 F1. This indicates that while progress has been made in temporal relation extraction, the complete pipeline for end-to-end timeline construction from raw EHRs is still an active research area.
Limitations and Future Work
- Scope of Treatments: The current shared task is limited to chemotherapy treatments. Future work could extend to other types of cancer therapies like immunotherapy, radiation therapy, and surgery.
- LLM Performance Gap: The paper explicitly notes that the performance gap between prompting LLMs and fine-tuning smaller LMs suggests a need for more sophisticated LLMs or prompting techniques to achieve optimal results in challenging clinical NLP tasks.
- Data Access: Access to the UPMC EHR data required a Data Use Agreement, a process that took several weeks, potentially limiting participation.
The authors express gratitude to annotators, domain experts, and funding agencies. They also emphasize the importance of ethical data handling, including de-identification and secure data sharing via Globus.
Summary updated