Deep sight: enhancing periprocedural adverse event recording in endoscopy by structuring text documentation with privacy-preserving large language models

Isabella C. Wiest, Dyke Ferber, Stefan Wittlinger, Matthias P. Ebert, Sebastian Belle, Jakob Nikolas KatheriGIE 2024 · DOI

TL;DR

This paper explores using Large Language Models (LLMs) to automatically extract adverse event (AE) information from unstructured endoscopy reports, aiming to improve documentation accuracy and efficiency. They found that privacy-preserving, locally deployable LLMs like Llama-2 perform comparably to proprietary models like GPT-4, offering a viable solution for real-world clinical settings.

What Problem Does This Paper Address?

Accurate documentation of adverse events (AEs) during endoscopic procedures is crucial for patient safety, quality control, and research. However, current practice heavily relies on free-text notes, which are difficult to analyze systematically. This unstructured format hinders scalable analysis, early detection of potential issues, and benchmarking of care quality. While LLMs offer a promising solution for extracting structured data, concerns about data privacy and the need for efficient, local deployment in clinical settings remain significant challenges.

Methods

The researchers developed an automated LLM pipeline to process endoscopy reports. They utilized two main LLM families: OpenAI's GPT-4 as a state-of-the-art benchmark and various Llama-2 models, including those fine-tuned for German, to assess privacy-preserving, locally deployable alternatives. The pipeline converted free-text reports into JavaScript Object Notation (JSON) for structured analysis. Prompt engineering was a key aspect, with experiments conducted to optimize the instructions given to the LLMs to improve their accuracy in identifying AEs like bleeding, perforation, and aspiration.

Datasets

The study used two datasets of endoscopy reports:

  • Proof-of-Concept Set (PoC-S): 171 reports collected from January 2022 to December 2022, used for initial prompt engineering and model exploration.
  • Test Set (TS): 501 colonoscopy reports collected from March 2010 to July 2012, used for final validation and performance evaluation.

The ground truth for AEs was established by three blinded medical experts, with conflicting cases resolved through discussion.

Key Insights and Ideas

  • Privacy-Preserving LLMs are Viable: Locally deployable LLMs, particularly Llama-2 models fine-tuned for German, demonstrate performance comparable to proprietary models like GPT-4, making them suitable for real-world clinical use where data privacy is paramount.
  • Prompt Engineering is Crucial: The effectiveness of LLMs in extracting medical information is highly sensitive to the prompt design. Tailoring prompts with specific instructions and examples can significantly improve both sensitivity and specificity.
  • Structured Data Revolutionizes Analysis: Moving from free-text to structured data (like JSON) via LLMs unlocks scalable analysis, immediate quality reporting, and reduces manual documentation burdens.
  • LLMs Can Catch Nuances: LLMs can interpret complex medical text, including identifying AEs even when there are minor errors or unclear descriptions in the original reports, outperforming simple keyword searches.

Key Results

  • GPT-4 Performance: Achieved high accuracy with a sensitivity of 97% and specificity of 92% on the PoC-S, and 91% sensitivity and 96% specificity on the test set.
  • Llama-2 Performance: German-fine-tuned Llama-2 models showed comparable results, with the "Sauerkraut" 70b model achieving 94% sensitivity and 92% specificity on the PoC-S, and 89% sensitivity and 93% specificity on the test set.
  • Prompt Engineering Impact: Experiments showed that specific prompt features could substantially influence specificity and sensitivity, with a prompt including an effort-based reward ("Make an effort, because you will get a lot of money for the right answer") yielding the best results (94% sensitivity, 94% specificity).
  • Keyword Search vs. LLM: LLMs significantly outperformed a simple keyword search for AEs, achieving much higher sensitivity and specificity (e.g., 84.5% sensitivity and 96.6% specificity for bleeding with LLM vs. 74.1% sensitivity and 85.7% specificity with keyword search).

Limitations and Future Work

  • Binary AE Assessment: The study primarily focused on binary AE detection (present/absent). Future work should explore extracting other categorical variables and more complex AE descriptions.
  • Prompt Engineering Complexity: While prompt engineering is effective, the exact mechanisms driving its success require further research.
  • Real-World Integration: Validation of LLM implementation and integration into existing clinical information systems is essential.
  • LLM Hallucinations: The potential for LLMs to invent data or misinterpret inputs needs ongoing monitoring and mitigation strategies.
  • Generalist LLMs: The study demonstrated that generalist LLMs can achieve near-perfect performance on medical information extraction, but further research into their limitations and regulatory considerations is needed.

Summary updated

Abstract

Applying LLMs to extract structured medical information, particularly from endoscopy reports, offers an efficient, scalable, and adaptable documentation method that captures adverse events accurately with a low error rate. It facilitates immediate quality reporting and reduces manual documentation efforts.
BibTeX
@inproceedings{wiest2024deep,
  author = {Isabella C. Wiest and Dyke Ferber and Stefan Wittlinger and Matthias P. Ebert and Sebastian Belle and Jakob Nikolas Kather},
  title = {{Deep sight: enhancing periprocedural adverse event recording in endoscopy by structuring text documentation with privacy-preserving large language models}},
  booktitle = {iGIE},
  year = {2024},
  doi = {10.1016/j.igie.2024.08.001},
}
Figure