Vision-language models for automated video analysis and documentation in laparoscopic surgery: a proof-of-concept study

Esther Helene Stueker, Fiona R. Kolbinger, Oliver Lester Saldanha, David Digomann, Steffen Pistorius, Florian Oehme, Marko van Treeck, Dyke Ferber, Chiara Maria Lavinia Löffler, Jürgen Weitz, Marius Distler, Jakob Nikolas Kather, Hannah Sophie MutiInternational Journal of Surgery 2025 · DOI

TL;DR

This study explores the potential of general-purpose Vision-Language Models (VLMs) like GPT-4o and Gemini-1.5-pro for automating surgical video analysis and documentation in laparoscopic procedures. While these models show promise in identifying surgical objects and classifying procedures, they struggle with more complex tasks like pathology grading and detailed step-by-step documentation, though performance can be improved with in-context learning.

What Problem Does This Paper Address?

The healthcare industry faces a significant shortage of medical personnel, leading to increased administrative burdens and a need for more efficient clinical documentation. Surgical documentation, in particular, is a time-consuming and repetitive task. Vision-Language Models (VLMs), which combine computer vision with natural language processing, offer a promising avenue to automate aspects of surgical video analysis and documentation, potentially freeing up valuable clinician time and improving the consistency of records. This paper investigates the capabilities of current, widely available VLMs for these tasks in the context of laparoscopic surgery.

Methods

The researchers conducted a comparative performance study using two leading general-purpose VLMs: GPT-4o (OpenAI) and Gemini-1.5-pro (Google). They developed a workflow where video frames from laparoscopic surgeries were processed by the VLMs, guided by system and user prompts. Four increasingly complex tasks were evaluated:

  1. Object Detection: Identifying surgical tools (vessel clips, gauze, retrieval bags) and active bleeding.
  2. Surgery Type Classification: Distinguishing between cholecystectomy (gallbladder removal) and appendectomy (appendix removal).
  3. Appendicitis Grading: Classifying the severity of appendicitis based on a predefined grading system.
  4. Surgical Report Generation: Creating descriptive surgical reports for the procedures.

To enhance performance, particularly for object recognition, the study also explored the impact of in-context learning (ICL), where additional examples or guidance were provided to the models. Performance was assessed using descriptive accuracy metrics.

Datasets

The study utilized two datasets:

  • CholecT45: A dataset of cholecystectomy videos.
  • LapApp: A dataset of appendectomy videos.

Both datasets were pre-processed to extract frames at a rate of 1 frame per second (fps) and resized. The study did not introduce a new benchmark dataset but rather evaluated existing VLMs on these established surgical video collections.

Key Insights and Ideas

  • VLMs are surprisingly capable out-of-the-box: Even general-purpose VLMs can perform basic surgical vision tasks with impressive accuracy without specific fine-tuning.
  • Object detection is a strong suit: Identifying common surgical instruments and events like bleeding is a relatively straightforward task for these models.
  • Complexity matters: As tasks become more nuanced (e.g., grading pathology, describing intricate procedural steps), VLM performance degrades significantly.
  • ICL is a powerful enhancer: Providing models with examples and structured guidance (in-context learning) can substantially improve their accuracy on challenging sub-tasks.
  • Domain-agnostic vs. Domain-specific: The study highlights the potential of general VLMs but also hints at the future need for models with specialized surgical knowledge for optimal performance.

Key Results

  • Object Detection: Both models achieved 100% accuracy for vessel clip detection. GPT-4o outperformed Gemini-1.5-pro in detecting retrieval bags (100% vs. 93.3%) and gauze (93.3% vs. 60%). Gemini-1.5-pro was slightly better at detecting bleeding (93.3% vs. 86.7%).
  • Surgery Identification: Gemini-1.5-pro was more accurate in classifying cholecystectomies (93% vs. 80%), while both models achieved 60% accuracy for appendectomies.
  • Appendicitis Grading: This was a weak point for both models, with GPT-4o achieving 40% accuracy and Gemini-1.5-pro achieving 26.7%.
  • Surgical Reports: GPT-40 produced more complete outputs for cholecystectomies (90.4%), while Gemini-1.5-pro showed higher overall correctness for both procedures (CCE: 71.1%, APE: 69.6%).
  • ICL Impact: In-context learning significantly improved tool recognition in appendectomies, for example, boosting GPT-40's performance from 69.2% to 80% for a specific step. However, its effect on organ removal step recognition was inconsistent.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Scope: The study only included two types of surgeries (cholecystectomy and appendectomy), limiting the generalizability of findings.
  • Data Leakage Concerns: The potential for data leakage from CCE videos into VLM training data was a concern.
  • Model Constraints: The need to reduce frame rates to 1 fps and the occasional exceeding of context windows by models like GPT-40 limited the depth of analysis.
  • Video Blocking: Gemini-1.5-pro blocked some videos due to content policies, which could affect performance.
  • Reproducibility: The inherent non-determinism of LLMs makes exact replication challenging.

Future work should focus on:

  • Comparing general-purpose VLMs with domain-specific VLMs trained on surgical data.
  • Expanding the evaluation to a broader range of minimally invasive surgeries.
  • Investigating more advanced ICL techniques and other methods to enhance performance on complex tasks.
  • Exploring the integration of VLMs as "companions" in the operating room, assisting surgeons with documentation and potentially decision support, while maintaining physician oversight.

Summary updated

Abstract

GPT-4o and Gemini-1.5-pro performed reliably in object detection and procedure classification but showed limitations in grading pathology and accurately describing procedural steps, which could be enhanced through in-context learning. This shows that domain-agnostic VLMs can be applied to surgical video analysis. In the future, VLMs with domain knowledge can be envisioned to enhance the operating room in the form of companions.
BibTeX
@article{stueker2025vision,
  author = {Esther Helene Stueker and Fiona R. Kolbinger and Oliver Lester Saldanha and David Digomann and Steffen Pistorius and Florian Oehme and Marko van Treeck and Dyke Ferber and Chiara Maria Lavinia Löffler and Jürgen Weitz and Marius Distler and Jakob Nikolas Kather and Hannah Sophie Muti},
  title = {{Vision-language models for automated video analysis and documentation in laparoscopic surgery: a proof-of-concept study}},
  journal = {International Journal of Surgery},
  year = {2025},
  doi = {10.1097/js9.0000000000003069},
}
Figure