Lexicans at Chemotimelines 2024: Chemotimeline Chronicles - Leveraging Large Language Models (LLMs) for Temporal Relations Extraction in Oncological Electronic Health Records

Vishakha Sharma, Andres Fernandez, Andrei Ioanovici, David Talby, Frederik BuijsNAACL 2024

TL;DR

This paper tackles the challenge of automatically constructing chemotherapy treatment timelines from Electronic Health Records (EHRs) using Large Language Models (LLMs). The authors demonstrate that LLMs like Llama 2 and Mistral 7B can achieve competitive results in extracting temporal relations between chemotherapy events and dates, paving the way for more efficient and accurate patient journey visualization in oncology.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) are a treasure trove of patient information, but their heterogeneous and unstructured nature makes it incredibly difficult to extract meaningful insights, especially for complex treatment journeys like those in oncology. Manually piecing together a patient's chemotherapy timeline from various notes, reports, and summaries is time-consuming and prone to errors. This gap hinders effective clinical decision-making and personalized treatment strategies. The Chemotimelines 2024 shared task aims to automate this process by building chemotherapy treatment timelines from EHR notes, a task this paper addresses.

Methods

The core of this work lies in leveraging state-of-the-art Large Language Models (LLMs) for temporal relation extraction. The authors employed a zero-shot prompting strategy, meaning they guided the LLMs to perform the task without specific fine-tuning on the target dataset.

Key methodological steps include:

  • Document Chunking: EHR documents were divided into manageable paragraphs or sections to facilitate processing.
  • Zero-Shot Prompting: Structured prompts were designed to guide LLMs (specifically Llama 2 and Mistral 7B) in identifying and extracting temporal relationships between chemotherapy events (drugs) and their associated dates. This involved prompt engineering to optimize the extraction of precise and relevant information.
  • Semantic Object Construction: Extracted information was structured into semantic-rich objects.
  • Directed Acyclic Graph (DAG): A simplified DAG was used to outline the logical framework for constructing the output taxonomy, ensuring data integrity and predictable model behavior.
  • Date Normalization: Temporal entities were normalized to absolute datetimes, handling various date formats and potential parsing failures.
  • Model Evaluation: The system was evaluated using standard metrics like Precision, Recall, and F-score.

Datasets

The study utilized the dataset provided for the Chemotimelines 2024 shared task. This dataset comprises 95 patients with 1438 reports across three indications: Breast, Melanoma, and Ovarian. The dataset includes gold annotations for chemotherapy events (EVENTs) and time expressions (TIMEX3s), along with temporal relation annotations based on the THYME ontology, specifically BEGINS-ON, CONTAINS, and ENDS-ON relations.

Key Insights and Ideas

  • LLMs as powerful tools for temporal relation extraction in EHRs: The study demonstrates that LLMs, even in a zero-shot setting, can effectively extract complex temporal relationships from unstructured clinical text.
  • Zero-shot prompting's effectiveness: The zero-shot approach, guided by well-engineered prompts, proved to be a viable strategy for relation extraction, reducing the need for extensive task-specific training data.
  • Importance of prompt engineering: The success of the zero-shot approach hinges on carefully crafting prompts that clearly instruct the LLM on the desired output format and the types of relations to identify.
  • Challenges in relation classification: The human evaluation revealed that LLMs sometimes misclassify temporal relations (e.g., mistaking CONTAINS for BEGINS-ON or ENDS-ON) and occasionally miss relations altogether.
  • High recall, lower precision trade-off: The LLM-based approach achieved high recall, which is beneficial for tasks requiring high sensitivity, but this came at the cost of lower precision, suggesting a tendency to over-predict relations.

Key Results

The study achieved a 5th place ranking in Subtask 1 of the Chemotimelines 2024 shared task, with an average F1 score of 0.71 using the Llama 2 model. Llama 2 outperformed Mistral 7B, achieving an average F1 of 0.61. Specifically, Llama 2 secured the 5th position overall, 4th for Melanoma, and 7th for Breast and Ovarian indications. The error analysis indicated that LLMs sometimes misclassify CONTAINS relations and can struggle with dates, occasionally misinterpreting them as future dates.

Limitations and Future Work

The authors acknowledge several limitations:

  • Precision vs. Recall: The current approach, while achieving high recall, suffers from lower precision, indicating a need for better discrimination between true and false positive relations.
  • LLM Biases: LLMs can perpetuate biases present in their training data, potentially leading to unfair or inaccurate timelines, especially across diverse patient populations.
  • Interpretability: The complexity of LLMs makes it challenging to interpret their decision-making processes, limiting clinician trust and validation.
  • Computational Resources: Training and fine-tuning LLMs require significant computational resources.
  • Human Oversight: Despite automation, human oversight and validation remain essential to ensure the accuracy and relevance of generated timelines.

Future work suggested includes:

  • Domain-Specific LLMs: Exploring LLMs specifically tailored for biomedical texts (e.g., JSL-MedMNX-7B) for potentially improved accuracy.
  • Further Training and Fine-tuning: Investigating more advanced training strategies, including human-in-the-loop models, to strike a better balance between automation and precision.
  • In-depth Error Analysis: Conducting more detailed error analyses to pinpoint root causes of false positives and identify patterns for targeted improvements.
  • Validation Across Diverse Datasets: Extending the evaluation to other healthcare datasets to assess the system's generalizability.

Summary updated

BibTeX
@inproceedings{sharma2024lexicans,
  author = {Vishakha Sharma and Andres Fernandez and Andrei Ioanovici and David Talby and Frederik Buijs},
  title = {{Lexicans at Chemotimelines 2024: Chemotimeline Chronicles - Leveraging Large Language Models (LLMs) for Temporal Relations Extraction in Oncological Electronic Health Records}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure