LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Lecture Learning

Jiajie Li, Garrett Skinner, Brian R Quaranto, Gene Yang, Steven D Schwaitzberg, Peter C W Kim, Jinjun XiongSubmission to ICLR 2025 · Rejected

TL;DR

LLaVA-Surg is an open-source multimodal conversational assistant designed specifically for understanding dynamic surgical videos. To overcome the scarcity of expert surgical data, the authors introduce a two-stage automated curation pipeline that converts online surgical lectures into Surg-QA—a dataset of over 102,000 video-instruction pairs spanning multiple procedural tiers from tool detection to expert reasoning and planning.

What Problem Does This Paper Address?

While multimodal large language models (MLLMs) such as LLaVA-Med have advanced clinical understanding of static medical images (e.g., chest X-rays, MRI slices), they fall short in surgery. Operative procedures are fundamentally dynamic, featuring intricate temporal dependencies, instrument tracking, and multi-step tactical decision-making.

Existing surgical VQA benchmarks are largely limited to single-frame classification tasks or single-turn QA covering basic tools and organs. The bottleneck has been data: curating dense, expert-annotated surgical video conversations is prohibitively expensive, while generic end-to-end LLM data generation pipelines frequently hallucinate clinical facts when confronted with long transcripts.

Methods

The authors introduce both an automated curation pipeline and a surgical video-language assistant:

  1. Two-Stage Structured Lecture Learning Pipeline:
  2. Stage 1 (Information Extraction): Raw surgical lecture voiceovers (from WebSurg) are transcribed using WhisperX. Instead of prompting an LLM to generate Q&A directly from noisy transcripts, Llama-3-70B-Instruct extracts structured surgical primitives following a four-level hierarchy: Observations ($I_o$), Reasons ($I_r$), Plans ($I_p$), and Deductions ($I_d$).
  3. Stage 2 (Q&A Generation): The structured primitives are used to generate multi-turn dialogues. By isolating the facts into structured slots first, hallucination risks during Q&A synthesis are dramatically reduced.
  4. Visual Concept Alignment: Video clips from CholecT50 with triplet annotations ([instrument, verb, target]) are processed by Llama-3-70B to synthesize concise descriptions grounding fundamental actions and tools.
  5. LLaVA-Surg Architecture: Built upon the Video-ChatGPT framework, the model samples $N$ frames per clip, encodes them via a frozen CLIP ViT-L/14, and pools spatial and temporal features into a video-level representation before projecting them into an LLM backbone initialized from LLaVA-Med. Fine-tuning is performed on 8 $\times$ A100 GPUs in ~6 hours.
Untrimmed Video + Voiceover
           │
     WhisperX (STT)
           │
           ▼
[Stage 1] Llama-3-70B Extraction ──► {Observation, Reason, Plan, Deduction}
                                                     │
[Stage 2] Llama-3-70B Q&A Gen    ◄───────────────────┘
           │
           ▼
  Surg-QA Dataset ──► Fine-tune LLaVA-Med Backbone + CLIP ──► LLaVA-Surg

Datasets

  • Surg-QA (New): Comprises 102,000 video-instruction pairs derived from 44,000 clips across 2,201 distinct surgical procedures (e.g., gastrointestinal, hepatobiliary, gynecologic, thoracic). It contains ~100K structured lecture pairs and ~2K visual concept alignment pairs.
  • CholecT50 (Existing): 50 endoscopic videos with frame-level action triplet annotations, used for concept alignment.
  • WebSurg (Existing source): Source repository for the 2,151 surgical lectures utilized.

Key Insights and Ideas

  • Deconstructing the "Surgical Knowledge Pyramid": Surgical video comprehension is modeled across four levels: superficial observations (Levels 1 & 2: objects, actions) vs. clinical reasoning (Levels 3 & 4: underlying rationales, next-step deductions, and plans). Surg-QA is the first dataset to explicitly include Levels 3 and 4 at scale.
  • Two-Stage Extraction Defeats Hallucination: Directly asking an LLM to generate surgical conversation from transcripts causes it to invent clinical rationales (e.g., conflating why a suture is placed). Splitting the task into deterministic information extraction followed by conditioned dialogue generation ensures factual consistency without needing commercial API endpoints.
  • Evaluating Expert AI with GPT: The authors verified automated GPT-3.5 evaluation against scoring by two practicing clinicians across 60 surgical videos, finding strong agreement and validating GPT-based scoring as an effective proxy for expert assessment in this domain.

Key Results

LLaVA-Surg was evaluated on a held-out test split of Surg-QA (4,359 open-ended QA pairs) under zero-shot conditions, benchmarked against generic video-LLMs and biomedical single-image LLMs:

Model GPT Score (0–5) Accuracy@all Accuracy@1
Video-ChatGPT 1.04 0.098 0.172
LLaVA-Med (1st frame only) 1.30 0.123 0.211
Video-LLaVA 1.32 0.129 0.224
LLaVA-Surg (Ours) 2.45 0.308 0.545
  • Doubled Performance: LLaVA-Surg achieved a 2.45/5.00 score and 54.5% Accuracy@1 (matching at least one key observation), outperforming Video-LLaVA (1.32 score, 22.4% Accuracy@1).
  • Clinical Utility: Qualitative case studies showed LLaVA-Surg correctly identifying procedures (e.g., Roux-en-Y gastric bypass) and specific intraoperative management choices (e.g., closing Petersen's space with non-absorbable monofilament suture to avoid internal herniation), where baseline models hallucinated irrelevant anatomy (such as appendices).

Limitations and Future Work

  • Model Hallucinations: Despite architectural and data improvements, LLaVA-Surg still exhibits domain-specific hallucinations during complex video sequences.
  • Automated Data Labeling Noise: Surg-QA was synthesized using an automated pipeline; while filtered and structured, it lacks universal manual review by surgical specialists.
  • Future Work: The authors highlight the need for expert clinician-in-the-loop validation of generated instruction data to further ground reliability in safety-critical surgical applications.

Summary updated

Abstract

Multimodal large language models (LLMs) have achieved notable success across various domains, while research in the medical field has largely focused on unimodal images. Meanwhile, current general-domain multimodal models for videos still lack the capabilities to understand and engage in conversations about surgical videos. One major contributing factor is the absence of datasets in the surgical field. In this paper, we create a new dataset, Surg-QA, consisting of 102,000 surgical video-instruction pairs, the largest of its kind so far. To build such a dataset, we propose a novel two-stage question-answer generation pipeline with LLM to learn surgical knowledge in a structured manner from the publicly available surgical lecture videos. The pipeline breaks down the generation process into two stages to significantly reduce the task complexity, allowing us to use a more affordable, locally deployed open-source LLM than the premium paid LLM services. It also mitigates the risk of LLM hallucinations during question-answer generation, thereby enhancing the overall quality of the generated data. We further train LLaVA-Surg, a novel vision-language conversational assistant capable of answering open-ended questions about surgical videos, on this Surg-QA dataset, and conduct comprehensive evaluations on zero-shot surgical video question-answering tasks. We show that LLaVA-Surg significantly outperforms all previous general-domain models, demonstrating exceptional multimodal conversational skills in answering open-ended questions about surgical videos. We will release our code, model, and the instruction-tuning dataset.
BibTeX
@misc{li2024llava,
  author = {Jiajie Li and Garrett Skinner and Brian R Quaranto and Gene Yang and Steven D Schwaitzberg and Peter C W Kim and Jinjun Xiong},
  title = {{LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Lecture Learning}},
  year = {2024},
  eprint = {2408.07981},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.07981},
}
Figure