Large language models as a diagnostic support tool in neuropathology

Katherine Hewitt, Isabella C. Wiest, Zunamys I. Carrero, Laura Bejan, Thomas O Millner, Sebastian Brandner, Jakob Nikolas KatherThe Journal of Pathology Clinical Research 2024 · DOI

TL;DR

Large language models (LLMs) show promise as diagnostic aids in neuropathology, but their effectiveness hinges on integrating them with up-to-date clinical guidelines. When equipped with Retrieval-Augmented Generation (RAG), LLMs accurately diagnosed neuropathological tumor subtypes in 90% of challenging cases, paving the way for AI-assisted reporting.

What Problem Does This Paper Address?

Diagnosing central nervous system (CNS) tumors is a complex and evolving field. The World Health Organization (WHO) classification system for these tumors is updated frequently, incorporating not only traditional morphological features but also crucial genetic and epigenetic information. Keeping abreast of these changes is a significant challenge for even experienced neuropathologists. While LLMs have demonstrated an ability to process vast amounts of medical text, their practical application in the nuanced domain of neuropathology, particularly in adhering to the latest diagnostic standards, remained largely untested. This paper aims to bridge that gap by evaluating the diagnostic accuracy of leading LLMs in classifying adult-type diffuse gliomas based on free-text histopathology reports.

Methods

The researchers evaluated three prominent LLMs: ChatGPT-4o, Claude-3.5-sonnet, and Llama3-70b. They employed two distinct approaches:

  • Zero-shot: The LLMs were prompted to provide diagnoses based solely on their pre-existing training data, without any specific contextual information about the latest guidelines.
  • Retrieval-Augmented Generation (RAG): This approach involved providing the LLMs with the relevant sections of the latest WHO CNS tumor classification guidelines (specifically the 2021 fifth edition). RAG constrains the LLM's output to information provided by the user, aiming to mitigate issues like hallucinations and reliance on potentially outdated training data.

The models' diagnoses were then compared against an expert-generated "ground truth" for strict concordance with the WHO guidelines, considering morphological, molecular, and grading components.

Datasets

The study utilized a dataset of 30 artificial neuropathology cases. These cases were generated based on real data from the pseudononymised University College London Hospitals (UCLH) dataset. The cases were specifically designed to represent a comprehensive range of features encountered in adult-type diffuse gliomas, including astrocytoma, oligodendroglioma, and glioblastoma, with varying grades and molecular characteristics. This dataset was created to mimic real-world diagnostic challenges and was not intended as a public benchmark.

Key Insights and Ideas

  • RAG is Crucial for Accuracy: The most significant finding is that LLMs, when used in a "zero-shot" manner, perform poorly in accurately diagnosing neuropathological cases according to current standards. Their performance dramatically improves when integrated with up-to-date clinical guidelines via RAG.
  • Outdated Terminology is a Major Hurdle: A primary reason for incorrect diagnoses in the zero-shot setting was the LLMs' tendency to use outdated terminology (e.g., "anaplastic") or incorrect formatting (e.g., Roman numerals for grades), which are no longer accepted by the latest WHO guidelines.
  • LLMs Can Mimic Expert Nuance (with RAG): The RAG approach allowed LLMs to not only identify the correct tumor type but also to adhere to the specific grading and molecular classifications mandated by the WHO, demonstrating their potential to assist in nuanced diagnostic reporting.
  • Potential for Diagnostic Support: The study strongly suggests that LLMs, particularly when augmented with RAG, can serve as valuable decision support tools for neuropathologists, helping them navigate complex and evolving diagnostic criteria.

Key Results

  • Zero-shot Performance: In the zero-shot experiments, the LLMs achieved a low overall correct diagnosis rate of 22.2% (20/90 cases). Claude-3.5-sonnet performed best, with 16 correct diagnoses, while ChatGPT-4o had 4, and Llama3-70b had none.
  • RAG Performance: With the RAG approach, the LLMs achieved a significantly higher accuracy rate of 90% (81/90 cases).
  • ChatGPT with RAG was the most accurate, with 28/30 correct diagnoses.
  • Claude-3.5-sonnet achieved 27/30 correct diagnoses.
  • Llama3-70b achieved 26/30 correct diagnoses.
  • Specific Tumor Types: Across RAG experiments, accuracy rates for astrocytoma, glioblastoma, and oligodendroglioma were all high, generally above 85% for each model.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Scope: The study focused exclusively on adult-type diffuse gliomas. Neuropathology encompasses a much wider range of entities, including rare cases and those requiring differential diagnoses based on exclusion criteria. Future work should explore LLMs' performance on these more diverse and challenging cases.
  • Data Sources: The study used a curated dataset of artificial cases. Evaluating LLMs on a broader array of data sources, including real-world, diverse, and potentially ambiguous cases, is necessary.
  • Confidentiality and Security: The paper highlights the critical need to address patient data confidentiality and security when deploying AI models in clinical settings. This includes exploring privacy-preserving techniques and secure cloud-based solutions.

The authors suggest that further prompt engineering and integration with RAG could enhance LLMs' utility for neuropathologists, potentially leading to a new generation of AI-assisted reporting tools.

Summary updated

Abstract

The WHO guidelines for classifying central nervous system (CNS) tumours are changing considerably with each release. The classification of CNS tumours is uniquely complex among most other solid tumours as it incorporates not just morphology, but also genetic and epigenetic features. Keeping current with these changes across medical fields can be challenging, even for clinical specialists. Large language models (LLMs) have demonstrated their ability to parse and process complex medical text, but their utility in neuro-oncology has not been systematically tested. We hypothesised that LLMs can effectively diagnose neuro-oncology cases from free-text histopathology reports according to the latest WHO guidelines. To test this hypothesis, we evaluated the performance of ChatGPT-4o, Claude-3.5-sonnet, and Llama3 across 30 challenging neuropathology cases, which each presented a complex mix of morphological and genetic information relevant to the diagnosis. Furthermore, we integrated these models with the latest WHO guidelines through Retrieval-Augmented Generation (RAG) and again assessed their diagnostic accuracy. Our data show that LLMs equipped with RAG, but not without RAG, can accurately diagnose the neuropathological tumour subtype in 90% of the tested cases. This study lays the groundwork for a new generation of computational tools that can assist neuropathologists in their daily reporting practice.
BibTeX
@article{hewitt2024large,
  author = {Katherine Hewitt and Isabella C. Wiest and Zunamys I. Carrero and Laura Bejan and Thomas O Millner and Sebastian Brandner and Jakob Nikolas Kather},
  title = {{Large language models as a diagnostic support tool in neuropathology}},
  journal = {The Journal of Pathology Clinical Research},
  year = {2024},
  doi = {10.1002/2056-4538.70009},
}
Figure