TL;DR
Large language models (LLMs) show promise as diagnostic aids in neuropathology, but their effectiveness hinges on integrating them with up-to-date clinical guidelines. When equipped with Retrieval-Augmented Generation (RAG), LLMs accurately diagnosed neuropathological tumor subtypes in 90% of challenging cases, paving the way for AI-assisted reporting.
What Problem Does This Paper Address?
Diagnosing central nervous system (CNS) tumors is a complex and evolving field. The World Health Organization (WHO) classification system for these tumors is updated frequently, incorporating not only traditional morphological features but also crucial genetic and epigenetic information. Keeping abreast of these changes is a significant challenge for even experienced neuropathologists. While LLMs have demonstrated an ability to process vast amounts of medical text, their practical application in the nuanced domain of neuropathology, particularly in adhering to the latest diagnostic standards, remained largely untested. This paper aims to bridge that gap by evaluating the diagnostic accuracy of leading LLMs in classifying adult-type diffuse gliomas based on free-text histopathology reports.
Methods
The researchers evaluated three prominent LLMs: ChatGPT-4o, Claude-3.5-sonnet, and Llama3-70b. They employed two distinct approaches:
- Zero-shot: The LLMs were prompted to provide diagnoses based solely on their pre-existing training data, without any specific contextual information about the latest guidelines.
- Retrieval-Augmented Generation (RAG): This approach involved providing the LLMs with the relevant sections of the latest WHO CNS tumor classification guidelines (specifically the 2021 fifth edition). RAG constrains the LLM's output to information provided by the user, aiming to mitigate issues like hallucinations and reliance on potentially outdated training data.
The models' diagnoses were then compared against an expert-generated "ground truth" for strict concordance with the WHO guidelines, considering morphological, molecular, and grading components.
Datasets
The study utilized a dataset of 30 artificial neuropathology cases. These cases were generated based on real data from the pseudononymised University College London Hospitals (UCLH) dataset. The cases were specifically designed to represent a comprehensive range of features encountered in adult-type diffuse gliomas, including astrocytoma, oligodendroglioma, and glioblastoma, with varying grades and molecular characteristics. This dataset was created to mimic real-world diagnostic challenges and was not intended as a public benchmark.
Key Insights and Ideas
- RAG is Crucial for Accuracy: The most significant finding is that LLMs, when used in a "zero-shot" manner, perform poorly in accurately diagnosing neuropathological cases according to current standards. Their performance dramatically improves when integrated with up-to-date clinical guidelines via RAG.
- Outdated Terminology is a Major Hurdle: A primary reason for incorrect diagnoses in the zero-shot setting was the LLMs' tendency to use outdated terminology (e.g., "anaplastic") or incorrect formatting (e.g., Roman numerals for grades), which are no longer accepted by the latest WHO guidelines.
- LLMs Can Mimic Expert Nuance (with RAG): The RAG approach allowed LLMs to not only identify the correct tumor type but also to adhere to the specific grading and molecular classifications mandated by the WHO, demonstrating their potential to assist in nuanced diagnostic reporting.
- Potential for Diagnostic Support: The study strongly suggests that LLMs, particularly when augmented with RAG, can serve as valuable decision support tools for neuropathologists, helping them navigate complex and evolving diagnostic criteria.
Key Results
- Zero-shot Performance: In the zero-shot experiments, the LLMs achieved a low overall correct diagnosis rate of 22.2% (20/90 cases). Claude-3.5-sonnet performed best, with 16 correct diagnoses, while ChatGPT-4o had 4, and Llama3-70b had none.
- RAG Performance: With the RAG approach, the LLMs achieved a significantly higher accuracy rate of 90% (81/90 cases).
- ChatGPT with RAG was the most accurate, with 28/30 correct diagnoses.
- Claude-3.5-sonnet achieved 27/30 correct diagnoses.
- Llama3-70b achieved 26/30 correct diagnoses.
- Specific Tumor Types: Across RAG experiments, accuracy rates for astrocytoma, glioblastoma, and oligodendroglioma were all high, generally above 85% for each model.
Limitations and Future Work
The authors acknowledge several limitations:
- Limited Scope: The study focused exclusively on adult-type diffuse gliomas. Neuropathology encompasses a much wider range of entities, including rare cases and those requiring differential diagnoses based on exclusion criteria. Future work should explore LLMs' performance on these more diverse and challenging cases.
- Data Sources: The study used a curated dataset of artificial cases. Evaluating LLMs on a broader array of data sources, including real-world, diverse, and potentially ambiguous cases, is necessary.
- Confidentiality and Security: The paper highlights the critical need to address patient data confidentiality and security when deploying AI models in clinical settings. This includes exploring privacy-preserving techniques and secure cloud-based solutions.
The authors suggest that further prompt engineering and integration with RAG could enhance LLMs' utility for neuropathologists, potentially leading to a new generation of AI-assisted reporting tools.
Summary updated