TL;DR
This paper introduces TimelyMed, an AI system that uses a locally deployed LLM to automatically extract and organize temporal and medical events from psychiatric medical records. By creating structured timelines with attribution to source text, TimelyMed aims to reduce clinicians' cognitive burden and improve decision-making in mental healthcare.
What Problem Does This Paper Address?
Understanding a patient's longitudinal clinical course is vital for effective psychiatric treatment. This involves tracking key events like symptom onset, episode recurrence, hospitalizations, and treatment responses. However, manually extracting and organizing this information from unstructured clinical notes is a time-consuming and error-prone process for physicians. Existing AI approaches, particularly large language models (LLMs), struggle with generating structured outputs and can be prone to hallucinations, making them unreliable for this critical task. The challenge lies in accurately identifying and temporalizing these events from diverse and often idiosyncratically written medical records.
Methods
TimelyMed employs a two-phase approach: offline training and online serving.
- Offline Training: A large language model (LLM) is fine-tuned using a supervised approach. The authors specifically mention using LoRA (Low-Rank Adaptation) for efficient fine-tuning. A crucial aspect of this phase is the design of a detailed system prompt that guides the LLM by specifying the task, event categories, desired output format (JSON), and providing examples (demonstrations). This prompt is key to ensuring the LLM generates structured and accurate outputs.
- Online Serving: This phase takes raw clinical notes and processes them in real-time.
- Sentence Segmentation: Clinical notes are first broken down into individual sentences.
- Event Extraction: A fine-tuned LLM then extracts both medical events (e.g., Symptom/Episode, Hospitalization, Remission/Response) and temporal events (e.g., Date, Age, Duration, Ago, Persistence).
- Temporal Alignment: Extracted temporal information is standardized into a consistent date format. A unique feature is the system's ability to find a relevant temporal event even if it's in a preceding sentence, ensuring medical events are properly anchored in time.
- Attribution: The system includes the original source sentences for each extracted event, providing transparency and allowing clinicians to verify the information.
Datasets
The study utilized the Integrated Medical Database of National Taiwan University Hospital (NTUH-iMD). Specifically, they focused on 500 discharge notes from the psychiatric unit, covering the period from January 1, 2006, to September 30, 2016. These notes were de-identified, and only sections related to the "history of the present illness" were used. An additional de-identification tool was applied, followed by manual review to ensure data integrity. The dataset was manually annotated by a board-certified clinical psychiatrist and an NLP researcher, categorizing sentences into "Symptom/Episode," "Hospitalization," "Remission/Response," or "None," and identifying temporal events.
Key Insights and Ideas
- Local LLM Deployment for Privacy: Using a locally deployed LLM is a critical design choice, ensuring the security and privacy of sensitive patient data, which is paramount in healthcare.
- Structured Output via Prompt Engineering: The success of the LLM in generating structured, attributable outputs hinges on a well-designed system prompt that clearly defines the task, categories, and format. This highlights the power of prompt engineering in adapting LLMs for specific, high-stakes applications.
- Attribution for Trust and Verification: Providing the original source sentence for each extracted event is a key feature that builds trust and allows clinicians to quickly verify the AI's findings, addressing the "black box" problem often associated with AI.
- Iterative Temporal Alignment: The ability to look back to preceding sentences for temporal context is an intelligent way to handle the often fragmented nature of clinical notes and ensure accurate timeline construction.
Key Results
The paper evaluated three models: LLaMA (Llama-3.1-8B-Instruct) and Mistral (Mistral-8B-Instruct-2410), with and without supervised fine-tuning (SFT).
- Medical Event Extraction: After fine-tuning, Mistral achieved the highest performance, with F1 scores of 0.9247 for Episode/Symptom and 0.9436 for Hospitalization. "Remission/Response" was the most challenging category, likely due to varied terminology.
- Temporal Event Extraction: Mistral also led in temporal event extraction, achieving F1 scores of 0.9677 for Age and 0.9334 for Ago. "Persistence" showed lower performance (F1 of 0.6892), attributed to its low occurrence in the dataset.
- The fine-tuned models (Llama and Mistral) significantly outperformed their non-fine-tuned counterparts across most categories.
Limitations and Future Work
- "Remission/Response" Category: The authors acknowledge that "Remission/Response" was the weakest performing category for medical event extraction, likely due to the diverse ways physicians describe symptom control.
- "Persistence" Temporal Event: The temporal event "Persistence" also showed lower performance, which the authors attribute to its infrequent occurrence in the annotated dataset.
- Future Work: The authors plan to focus on refining event classification and expanding the system's applicability to a wider range of psychiatric conditions. They also aim to further enhance the system's reliability and potentially explore more advanced LLM architectures or training techniques.
Summary updated