TL;DR
This paper explores using Meta's Llama 2, a powerful generative Large Language Model (LLM), to extract seizure frequency data from unstructured clinical notes in Electronic Health Records (EHRs). The study demonstrates that LLMs can effectively mine this complex data, offering a promising avenue for improving clinical NLP research in epilepsy and beyond.
What Problem Does This Paper Address?
Extracting precise seizure frequency from free-text clinical notes in Electronic Health Records (EHRs) is a significant challenge. These notes, often comprising doctors' and nurses' reports from outpatient visits, are unstructured and contain a mix of medical and administrative information. Crucially, they frequently lack explicit seizure frequency data, or the information provided is vague and imprecise. This makes it difficult to conduct research, such as investigating the associations between anti-seizure medications (ASMs) and seizure frequency, which is vital for treating epilepsy, especially in patients who don't respond to initial treatments. While previous research has used pre-trained Transformers for similar tasks, the application of newer, more powerful generative LLMs like Llama 2 to this specific problem remains largely unexplored.
Methods
The researchers employed Meta's Llama 2 (specifically a 13B parameter chat version, quantized by GPTQ) as their primary tool for extracting seizure frequency. They utilized the LangChain framework for model development and implementation. A key aspect of their methodology involved prompt engineering, where they found that "few-shot prompting" with 11 examples significantly improved Llama 2's ability to identify seizure frequency. They also discovered that instructing the model to act as a "professional neuroscientist" and provide "succinct answers" was crucial. Furthermore, structuring the query as a three-step logical process, rather than a single question, optimized the model's performance. To handle the nuances of seizure frequency, they initially defined nine categories, which were later aggregated into three broader categories (infrequent, frequent, and unknown) to improve accuracy. For comparison, they also fine-tuned BERT Large and RoBERTa Large models on a subset of the data.
Datasets
The study utilized a dataset of 41,340 EHRs from King's College Hospital (KCH) in London, spanning from 2013-2022. These records pertained to 6,853 unique adult patients with epilepsy. A subset of 3,000 EHRs was selected using stratified random sampling to create an annotated dataset. This annotated dataset was then used for evaluating Llama 2. For comparative analysis with BERT and RoBERTa, a balanced subset of 1,720 EHRs was created, with an 80:10:10 split for training, validation, and testing.
Key Insights and Ideas
- LLMs as powerful clinical information extractors: The study strongly suggests that generative LLMs like Llama 2 are not just capable of general language tasks but can be effectively applied to extract specific, nuanced clinical information from unstructured EHRs.
- Prompt engineering is paramount: The success of LLMs in this domain is heavily reliant on how they are prompted. Few-shot learning and carefully crafted instructions significantly outperform simpler approaches.
- Aggregation simplifies complexity: While initially defining nine granular seizure frequency categories, aggregating them into three broader categories (infrequent, frequent, unknown) led to more accurate and usable results for the broader research project. This highlights a practical approach to managing LLM output for specific downstream tasks.
- Llama 2's advantage over Transformers: Llama 2 demonstrated superior performance compared to BERT and RoBERTa, potentially due to its larger size and generative capabilities, which allow it to leverage its pre-training on vast general corpora for clinical tasks without extensive fine-tuning.
Key Results
- On the full annotated dataset of 3,000 EHRs, Llama 2 achieved an F1 score of 0.73 and an accuracy of 0.94. The F1 score is preferred as a more robust measure due to a high number of true negatives in the accuracy calculation.
- Llama 2 performed particularly well in identifying letters with no or ambiguous seizure frequency information, achieving an F1 score of 0.87.
- When aggregated into three categories, Llama 2 achieved F1 scores of 0.87 for "unknown," 0.62 for "frequent," and 0.30 for "infrequent" seizures.
- Compared to BERT and RoBERTa on a smaller test set, Llama 2 showed better performance, with an F1 score of 0.54 compared to BERT's 0.55 and RoBERTa's 0.58. However, it's noted that Llama 2's performance on the larger dataset is a more reliable indicator.
- Llama 2 consistently made predictions on all EHRs, whereas BERT and RoBERTa failed to make predictions on a significant percentage of test EHRs.
Limitations and Future Work
- Reproducibility: The confidential nature of the EHRs means the results are not directly reproducible by external researchers without access to the hospital's secure IT network.
- Limited LLM experimentation: Due to regulatory constraints and API access limitations, the researchers could not experiment with proprietary LLMs like ChatGPT.
- Inter-annotator agreement: The study design, where annotators worked on separate batches, prevented the measurement of inter-annotator agreement.
- Computational resources: The researchers were limited by their GPU platform, preventing them from using the larger 70B parameter version of Llama 2.
- Dataset specificity: The dataset from King's College Hospital might not be representative of epilepsy patients from other hospitals.
Future work could involve exploring other open-source LLMs, improving prompt strategies, and investigating how to optimize performance for less frequent seizure categories.
Summary updated