Evaluating Large Language Models (LLMs) in Information Extraction: A Case Study of Extracting Cognitive Exam Dates and Scores

Hao Zhang, Neil Jethani, Simon Jones, Nicholas Genes, Vincent J. Major, Ian Jaffe, Anthony B. Cardillo, Noah Heilenbach, Nadia Fazal Ali, Luke J. Bonanni, Andrew J. Clayburn, Zain Khera, Erica C. Sadler, Jaideep Prasad, Jamie Schlacter, Kevin Liu, Benjamin A. Silva, Sophie Montgomery, Eric J. Kim, Jacob Lester, Theodore M. Hill, Alba Avoricani, Ethan Chervonski, James Davydov, William Small, Eesha Chakravartty, Himanshu Grover, John A. Dodson, Abraham A. Brody, Yindalon Aphinyanaphongs, Arjun V. Masurkar, Narges RazavianAlzheimer 2024 · DOI

TL;DR

This paper evaluates the performance of two leading Large Language Models (LLMs), ChatGPT (GPT-4) and Llama2, in extracting crucial cognitive assessment data (MMSE and CDR scores and dates) from clinical notes. The findings highlight ChatGPT's superior accuracy, suggesting LLMs hold promise for streamlining dementia research and clinical workflows, but underscore the critical need for rigorous validation.

What Problem Does This Paper Address?

Extracting specific, structured information from unstructured clinical notes is a time-consuming and labor-intensive task for researchers and clinicians. This is particularly true for identifying key data points related to cognitive assessments, such as scores from the Mini Mental State Exam (MMSE) and the Clinical Dementia Rating (CDR), along with the dates these assessments were performed. Such data is vital for tracking disease progression, identifying eligible patients for clinical trials, and informing treatment decisions. While Large Language Models (LLMs) offer powerful natural language processing capabilities, their reliability in extracting precise medical information needs thorough evaluation to ensure accuracy and avoid potentially harmful false results. This paper addresses the gap in understanding LLM performance for this specific, high-stakes medical information extraction task.

Methods

The researchers compiled a dataset of 765 clinical notes that mentioned MMSE or CDR. To establish a ground truth, 22 medically trained experts manually annotated these notes. Two state-of-the-art LLMs, ChatGPT (GPT-4, version "2023-03-15-preview") and Llama2 ("Llama-2-70b-chat"), were then employed to automatically extract MMSE and CDR instances along with their corresponding dates. The LLMs successfully processed 742 of the notes. A subset of 20 notes was used for fine-tuning and training the human reviewers. The remaining 722 notes were assigned to reviewers, with 309 notes being reviewed by two experts simultaneously to assess inter-rater reliability and identify discrepancies. Standard metrics such as precision, sensitivity, true/false negative rates, and overall accuracy were calculated. For the double-reviewed notes, a qualitative analysis of errors was also conducted.

Datasets

The study utilized a custom dataset of 765 clinical notes that contained mentions of the MMSE or CDR. This dataset was manually annotated by 22 medical experts to create a ground truth for evaluating LLM performance. No pre-existing public benchmark datasets for this specific task were mentioned.

Key Insights and Ideas

  • LLMs can automate tedious data extraction: The core idea is that LLMs can significantly reduce the manual effort required to pull specific data points from large volumes of clinical text.
  • Performance varies significantly between LLMs: Not all LLMs are created equal when it comes to medical data. The study clearly shows a substantial difference in accuracy between ChatGPT and Llama2 for this task.
  • MMSE extraction is more robust than CDR: The models performed much better on MMSE data than on CDR data, suggesting that the structure or common phrasing of MMSE mentions might be easier for LLMs to parse.
  • Qualitative error analysis is crucial: Simply looking at accuracy numbers isn't enough. Understanding why an LLM makes a mistake (e.g., hallucination, misinterpreting scores, wrong dates) is vital for improving them and trusting their outputs.

Key Results

For MMSE information extraction:

  • ChatGPT achieved an accuracy of 83%, sensitivity of 89.7%, true-negative rate of 96%, and precision of 82.7%.
  • Llama2 achieved an accuracy of 66.4%, sensitivity of 69.9%, true-negative rate of 60.0%, and precision of 62.2%.

For CDR information extraction:

  • ChatGPT achieved an accuracy of 87.1%, sensitivity of 84.3%, true-negative rate of 99.8%, and precision of 48.3%.
  • Llama2 achieved an accuracy of 74.5%, sensitivity of 39.7%, true-negative rate of 98.4%, and precision of 16.1%.

Qualitatively, Llama2 exhibited more severe errors, including 27 instances of total hallucination and 25 missed scores. ChatGPT, while not perfect, had significantly fewer critical errors.

Limitations and Future Work

The authors acknowledge that rigorous evaluation of LLMs is crucial for understanding their capabilities and limitations. While ChatGPT demonstrated promising results, the study's findings are specific to the versions of the models tested and the particular dataset used. Future work could involve testing newer LLM versions, expanding the dataset to include a wider variety of clinical notes and cognitive assessments, and exploring methods for further improving LLM accuracy and robustness, particularly for more complex tasks like CDR extraction. Developing techniques to mitigate specific error types, such as hallucination and date misinterpretation, would also be valuable.

Summary updated

Abstract

Abstract Background Large language models (LLMs) provide powerful natural language processing capabilities in medical and clinical tasks. Evaluating LLM performance is crucial due to potential false results. In this study, we assessed ChatGPT and Llama2, two state‐of‐the‐art LLMs, in extracting information from clinical notes, focusing on cognitive tests, specifically the Mini Mental State Exam (MMSE) and Cognitive Dementia Rating (CDR). Method We compiled a dataset consisting of 765 clinical notes mentioning MMSE and CDR. 22 medically trained experts provided the ground truth. ChatGPT (GPT‐4, version “2023‐03‐15‐preview”) and Llama2 (“Llama‐2‐70b‐chat”) were used to extract MMSE and CDR instances with corresponding dates. Inference was successful for 742 notes. We used 20 notes for fine‐tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 assigned to two reviewers simultaneously. Precision, sensitivity, true/false negative rates, and accuracy were calculated. For double‐reviewed notes, we qualitatively assessed the errors. Result The patient and note characteristics can be found in Table 1. For MMSE information extraction, ChatGPT (vs. Llama2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true‐negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true‐negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and Llama2 on double‐reviewed notes. Llama2 errors included 27 cases of total hallucination, 19 cases where other scores were reported instead of MMSE, 25 missed scores, and 23 cases where the wrong date was reported for the right score. In comparison, ChatGPT’s errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusion ChatGPT exhibited high accuracy in extracting MMSE scores and dates, with better performance compared to Llama2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.
BibTeX
@inproceedings{zhang2024evaluating,
  author = {Hao Zhang and Neil Jethani and Simon Jones and Nicholas Genes and Vincent J. Major and Ian Jaffe and Anthony B. Cardillo and Noah Heilenbach and Nadia Fazal Ali and Luke J. Bonanni and Andrew J. Clayburn and Zain Khera and Erica C. Sadler and Jaideep Prasad and Jamie Schlacter and Kevin Liu and Benjamin A. Silva and Sophie Montgomery and Eric J. Kim and Jacob Lester and Theodore M. Hill and Theodore M. Hill and Alba Avoricani and Ethan Chervonski and James Davydov and William Small and Eesha Chakravartty and Himanshu Grover and John A. Dodson and Abraham A. Brody and Yindalon Aphinyanaphongs and Arjun V. Masurkar and Narges Razavian},
  title = {{Evaluating Large Language Models (LLMs) in Information Extraction: A Case Study of Extracting Cognitive Exam Dates and Scores}},
  booktitle = {Alzheimer},
  year = {2024},
  doi = {10.1002/alz.087416},
}
Figure