TL;DR
LLaVA-Surg is an open-source multimodal conversational assistant designed specifically for understanding dynamic surgical videos. To overcome the scarcity of expert surgical data, the authors introduce a two-stage automated curation pipeline that converts online surgical lectures into Surg-QA—a dataset of over 102,000 video-instruction pairs spanning multiple procedural tiers from tool detection to expert reasoning and planning.
What Problem Does This Paper Address?
While multimodal large language models (MLLMs) such as LLaVA-Med have advanced clinical understanding of static medical images (e.g., chest X-rays, MRI slices), they fall short in surgery. Operative procedures are fundamentally dynamic, featuring intricate temporal dependencies, instrument tracking, and multi-step tactical decision-making.
Existing surgical VQA benchmarks are largely limited to single-frame classification tasks or single-turn QA covering basic tools and organs. The bottleneck has been data: curating dense, expert-annotated surgical video conversations is prohibitively expensive, while generic end-to-end LLM data generation pipelines frequently hallucinate clinical facts when confronted with long transcripts.
Methods
The authors introduce both an automated curation pipeline and a surgical video-language assistant:
- Two-Stage Structured Lecture Learning Pipeline:
- Stage 1 (Information Extraction): Raw surgical lecture voiceovers (from WebSurg) are transcribed using WhisperX. Instead of prompting an LLM to generate Q&A directly from noisy transcripts, Llama-3-70B-Instruct extracts structured surgical primitives following a four-level hierarchy: Observations ($I_o$), Reasons ($I_r$), Plans ($I_p$), and Deductions ($I_d$).
- Stage 2 (Q&A Generation): The structured primitives are used to generate multi-turn dialogues. By isolating the facts into structured slots first, hallucination risks during Q&A synthesis are dramatically reduced.
- Visual Concept Alignment: Video clips from CholecT50 with triplet annotations (
[instrument, verb, target]) are processed by Llama-3-70B to synthesize concise descriptions grounding fundamental actions and tools. - LLaVA-Surg Architecture: Built upon the Video-ChatGPT framework, the model samples $N$ frames per clip, encodes them via a frozen CLIP ViT-L/14, and pools spatial and temporal features into a video-level representation before projecting them into an LLM backbone initialized from LLaVA-Med. Fine-tuning is performed on 8 $\times$ A100 GPUs in ~6 hours.
Untrimmed Video + Voiceover
│
WhisperX (STT)
│
▼
[Stage 1] Llama-3-70B Extraction ──► {Observation, Reason, Plan, Deduction}
│
[Stage 2] Llama-3-70B Q&A Gen ◄───────────────────┘
│
▼
Surg-QA Dataset ──► Fine-tune LLaVA-Med Backbone + CLIP ──► LLaVA-Surg
Datasets
- Surg-QA (New): Comprises 102,000 video-instruction pairs derived from 44,000 clips across 2,201 distinct surgical procedures (e.g., gastrointestinal, hepatobiliary, gynecologic, thoracic). It contains ~100K structured lecture pairs and ~2K visual concept alignment pairs.
- CholecT50 (Existing): 50 endoscopic videos with frame-level action triplet annotations, used for concept alignment.
- WebSurg (Existing source): Source repository for the 2,151 surgical lectures utilized.
Key Insights and Ideas
- Deconstructing the "Surgical Knowledge Pyramid": Surgical video comprehension is modeled across four levels: superficial observations (Levels 1 & 2: objects, actions) vs. clinical reasoning (Levels 3 & 4: underlying rationales, next-step deductions, and plans). Surg-QA is the first dataset to explicitly include Levels 3 and 4 at scale.
- Two-Stage Extraction Defeats Hallucination: Directly asking an LLM to generate surgical conversation from transcripts causes it to invent clinical rationales (e.g., conflating why a suture is placed). Splitting the task into deterministic information extraction followed by conditioned dialogue generation ensures factual consistency without needing commercial API endpoints.
- Evaluating Expert AI with GPT: The authors verified automated GPT-3.5 evaluation against scoring by two practicing clinicians across 60 surgical videos, finding strong agreement and validating GPT-based scoring as an effective proxy for expert assessment in this domain.
Key Results
LLaVA-Surg was evaluated on a held-out test split of Surg-QA (4,359 open-ended QA pairs) under zero-shot conditions, benchmarked against generic video-LLMs and biomedical single-image LLMs:
| Model | GPT Score (0–5) | Accuracy@all | Accuracy@1 |
|---|---|---|---|
| Video-ChatGPT | 1.04 | 0.098 | 0.172 |
| LLaVA-Med (1st frame only) | 1.30 | 0.123 | 0.211 |
| Video-LLaVA | 1.32 | 0.129 | 0.224 |
| LLaVA-Surg (Ours) | 2.45 | 0.308 | 0.545 |
- Doubled Performance: LLaVA-Surg achieved a 2.45/5.00 score and 54.5% Accuracy@1 (matching at least one key observation), outperforming Video-LLaVA (1.32 score, 22.4% Accuracy@1).
- Clinical Utility: Qualitative case studies showed LLaVA-Surg correctly identifying procedures (e.g., Roux-en-Y gastric bypass) and specific intraoperative management choices (e.g., closing Petersen's space with non-absorbable monofilament suture to avoid internal herniation), where baseline models hallucinated irrelevant anatomy (such as appendices).
Limitations and Future Work
- Model Hallucinations: Despite architectural and data improvements, LLaVA-Surg still exhibits domain-specific hallucinations during complex video sequences.
- Automated Data Labeling Noise: Surg-QA was synthesized using an automated pipeline; while filtered and structured, it lacks universal manual review by surgical specialists.
- Future Work: The authors highlight the need for expert clinician-in-the-loop validation of generated instruction data to further ground reliability in safety-critical surgical applications.
Summary updated