TL;DR
This paper explores using a locally hosted, open-weights Large Language Model (LLM), Mixtral 8x7B, to extract chemotherapy treatment timelines from clinical notes. The approach aims to interpret complex medical language without domain-specific fine-tuning, highlighting the potential of LLMs for specialized biomedical data processing.
What Problem Does This Paper Address?
Extracting structured information about chemotherapy treatment timelines from unstructured clinical narratives is a significant challenge in healthcare informatics. The complexity of medical language and the need to piece together information across multiple patient notes make this a difficult task. Traditional methods often rely on rule-based systems or machine learning models trained on extensive, domain-specific annotated data. This paper tackles the problem of extracting these crucial timelines, specifically for the Chemotimelines 2024 shared task, by investigating the capabilities of general-purpose LLMs in a specialized domain.
Methods
The NYULangone team's approach leverages a locally deployed instance of Mixtral 8x7B, an open-weights LLM. The system employs a two-round inference process:
- Event Extraction per Note: The first round uses a carefully crafted prompt to instruct Mixtral to read individual patient notes and extract chemotherapy events. Each event is defined by the drug name, a temporal relation (e.g., "begins-on," "ends-on," "contains-1"), and a date in YYYY-MM-DD format. The prompt emphasizes adherence to THYME guidelines and requires output in a specific JSON format.
- Timeline Aggregation per Patient: The second round takes the extracted events from multiple notes for a single patient and uses another prompt to aggregate them into a comprehensive, patient-level chemotherapy timeline.
Crucially, the system does not use domain-specific fine-tuning. Instead, it relies on the pre-trained weights of the Mixtral LLM and processes raw text files, structuring the output into the required JSON format.
Datasets
The paper describes participation in the Chemotimelines 2024 shared task. While the specific training datasets used by the competition organizers are not detailed, the authors evaluated their system on the competition's development (dev) and validation sets. The competition itself focuses on extracting chemotherapy treatment timelines from clinical narratives.
Key Insights and Ideas
- Potential of Local, Open-Weights LLMs: The study demonstrates that even without domain-specific fine-tuning, a powerful local LLM like Mixtral can interpret and extract complex medical information, offering a promising direction for specialized NLP tasks.
- Prompt Engineering is Key: The success of the system hinges on carefully designed prompts that guide the LLM to understand its role, the task's requirements, and the desired output format.
- Two-Stage Processing: A multi-stage approach (per-note extraction followed by patient-level aggregation) is effective for handling information spread across multiple documents.
- Challenges of General LLMs in Specialized Domains: Despite their power, general LLMs still struggle with nuances in specialized fields, leading to errors like confabulating drugs or misclassifying non-chemotherapeutic agents.
Key Results
On the development set, the system achieved an average F1 score of 0.35. On the validation set, the average F1 score across different cancer types was 0.23. While these scores were below the baseline and leading teams, the paper highlights that the performance, though objectively poor, provided valuable insights. The authors note that local LLMs like Mixtral are still a new technology and currently perform at a level comparable to GPT-3 in general reasoning.
Limitations and Future Work
The authors acknowledge the system's performance was well below competitive benchmarks. They identify several types of errors:
- Confabulation: Extracting drugs not present in the text (e.g., from radiology reports).
- Inclusion of Non-Chemotherapeutic Drugs: Misidentifying drugs like steroids as chemotherapy agents.
- Failure to Extract Clearly Mentioned Drugs: Missing explicitly stated chemotherapy drugs.
Future work suggested includes enhancing the model's understanding of clinical contexts through retrieval-augmented generation (RAG) and exploring ensemble prompting techniques like "tree of thought." The authors believe that further research in these areas can improve the performance of local LLMs in specialized domains.
Summary updated