TL;DR
This paper evaluates the performance of two leading Large Language Models (LLMs), ChatGPT (GPT-4) and Llama2, in extracting crucial cognitive assessment data (MMSE and CDR scores and dates) from clinical notes. The findings highlight ChatGPT's superior accuracy, suggesting LLMs hold promise for streamlining dementia research and clinical workflows, but underscore the critical need for rigorous validation.
What Problem Does This Paper Address?
Extracting specific, structured information from unstructured clinical notes is a time-consuming and labor-intensive task for researchers and clinicians. This is particularly true for identifying key data points related to cognitive assessments, such as scores from the Mini Mental State Exam (MMSE) and the Clinical Dementia Rating (CDR), along with the dates these assessments were performed. Such data is vital for tracking disease progression, identifying eligible patients for clinical trials, and informing treatment decisions. While Large Language Models (LLMs) offer powerful natural language processing capabilities, their reliability in extracting precise medical information needs thorough evaluation to ensure accuracy and avoid potentially harmful false results. This paper addresses the gap in understanding LLM performance for this specific, high-stakes medical information extraction task.
Methods
The researchers compiled a dataset of 765 clinical notes that mentioned MMSE or CDR. To establish a ground truth, 22 medically trained experts manually annotated these notes. Two state-of-the-art LLMs, ChatGPT (GPT-4, version "2023-03-15-preview") and Llama2 ("Llama-2-70b-chat"), were then employed to automatically extract MMSE and CDR instances along with their corresponding dates. The LLMs successfully processed 742 of the notes. A subset of 20 notes was used for fine-tuning and training the human reviewers. The remaining 722 notes were assigned to reviewers, with 309 notes being reviewed by two experts simultaneously to assess inter-rater reliability and identify discrepancies. Standard metrics such as precision, sensitivity, true/false negative rates, and overall accuracy were calculated. For the double-reviewed notes, a qualitative analysis of errors was also conducted.
Datasets
The study utilized a custom dataset of 765 clinical notes that contained mentions of the MMSE or CDR. This dataset was manually annotated by 22 medical experts to create a ground truth for evaluating LLM performance. No pre-existing public benchmark datasets for this specific task were mentioned.
Key Insights and Ideas
- LLMs can automate tedious data extraction: The core idea is that LLMs can significantly reduce the manual effort required to pull specific data points from large volumes of clinical text.
- Performance varies significantly between LLMs: Not all LLMs are created equal when it comes to medical data. The study clearly shows a substantial difference in accuracy between ChatGPT and Llama2 for this task.
- MMSE extraction is more robust than CDR: The models performed much better on MMSE data than on CDR data, suggesting that the structure or common phrasing of MMSE mentions might be easier for LLMs to parse.
- Qualitative error analysis is crucial: Simply looking at accuracy numbers isn't enough. Understanding why an LLM makes a mistake (e.g., hallucination, misinterpreting scores, wrong dates) is vital for improving them and trusting their outputs.
Key Results
For MMSE information extraction:
- ChatGPT achieved an accuracy of 83%, sensitivity of 89.7%, true-negative rate of 96%, and precision of 82.7%.
- Llama2 achieved an accuracy of 66.4%, sensitivity of 69.9%, true-negative rate of 60.0%, and precision of 62.2%.
For CDR information extraction:
- ChatGPT achieved an accuracy of 87.1%, sensitivity of 84.3%, true-negative rate of 99.8%, and precision of 48.3%.
- Llama2 achieved an accuracy of 74.5%, sensitivity of 39.7%, true-negative rate of 98.4%, and precision of 16.1%.
Qualitatively, Llama2 exhibited more severe errors, including 27 instances of total hallucination and 25 missed scores. ChatGPT, while not perfect, had significantly fewer critical errors.
Limitations and Future Work
The authors acknowledge that rigorous evaluation of LLMs is crucial for understanding their capabilities and limitations. While ChatGPT demonstrated promising results, the study's findings are specific to the versions of the models tested and the particular dataset used. Future work could involve testing newer LLM versions, expanding the dataset to include a wider variety of clinical notes and cognitive assessments, and exploring methods for further improving LLM accuracy and robustness, particularly for more complex tasks like CDR extraction. Developing techniques to mitigate specific error types, such as hallucination and date misinterpretation, would also be valuable.
Summary updated