TL;DR
SurgQ-VLM is a new framework for surgical video question answering that tackles the "information bottleneck" of video compression. By using a question-conditioned dual-stream compressor and a reinforcement learning stage with rewards tailored to specific answer types (like counting or timing), it achieves a 10x BLEU-4 gain over zero-shot baselines and significantly outperforms existing surgical video models.
What Problem Does This Paper Address?
Reasoning about surgical video is inherently complex. To document a procedure or assess a surgeon's skill, an AI needs to know not just what is happening, but when it happens, in what order, and for how long.
Current Video Vision-Language Models (VLMs) face two major hurdles in this domain:
- Question-Agnostic Compression: Most models compress a 30-second video clip into a fixed number of tokens regardless of the question. If you ask about a split-second instrument change, a generic compressor might have already discarded that specific temporal evidence to save space.
- Structural Heterogeneity: Surgical answers aren't just simple labels. They span five incompatible structures: binary (yes/no), classification, multi-label (lists of tools), counting, and sentence-temporal (describing events with timestamps). Standard training signals (like cross-entropy) struggle to score these diverse formats uniformly well.
Methods
The authors propose SurgQ-VLM, a two-stage model based on a LoRA-adapted Qwen2.5-VL-7B.
1. Question-Conditioned Dual-Stream Compression
Instead of treating all video data the same, SurgQ-VLM uses two streams:
- Slow Stream (0.5 FPS): Captures high-spatial detail for identifying anatomy and phases.
- Fast Stream (1.0 FPS): Captures rapid instrument transitions and actions.
The "secret sauce" is the GuideInjector. As shown in the pipeline below, visual tokens from both streams are conditioned on the specific question via cross-attention before being fed to the LLM. This allows the model to "look for" the evidence required by the question during the compression process, shrinking 32,805 raw patches down to just 671 highly relevant tokens.
2. Answer-Type-Aware Reinforcement Learning
After initial Supervised Fine-Tuning (SFT), the model undergoes a second stage of training using Group Relative Policy Optimization (GRPO). Crucially, the reward function is "answer-type-aware." Rather than a single accuracy score, it dispatches different metrics based on the question's structure:
- Multi-label: Scored via micro-F1.
- Counting: Scored by normalized absolute error.
- Temporal: Scored by a blend of keyword accuracy and timing precision (within ±2 seconds).
Datasets
The authors introduced a massive new multi-task benchmark for surgical video QA:
- 80,636 QA pairs derived from the well-known Cholec80 and CholecT50 datasets (laparoscopic gallbladder removals).
- 18 question types covering phases, instruments, and complex "triplets" (instrument-verb-target).
- Data Split: Partitioned at the video level to ensure no surgery appears in both training and testing, preventing the model from simply memorizing specific procedures.
Key Insights and Ideas
- Conditioning is the Load-Bearer: The biggest performance drop occurred when question conditioning was removed. Without knowing the question, the compressor loses the specific temporal evidence needed for timing-related answers.
- Emergent Specialization: By zeroing out different token groups at inference, the authors found that the model naturally learned to use the "Fast" stream for temporal questions and the "Slow" stream for counting and spatial tasks.
- RL Closes the "Hard Type" Gap: While SFT is good for learning domain vocabulary, it plateaus on structurally difficult questions. The custom GRPO reward provided the necessary gradient signal to improve counting and temporal reasoning where standard cross-entropy failed.
Key Results
SurgQ-VLM sets a new state-of-the-art for surgical VQA:
- Accuracy: Reached 0.779 keyword accuracy, outperforming specialized models like SurgVidLM (0.740).
- Language Quality: Achieved a 10x BLEU-4 gain over zero-shot Qwen2.5-VL.
- GRPO Lift: The reinforcement learning stage added a 2.5 percentage point (pp) boost in accuracy, with the largest gains (+5.1 pp) seen in the difficult sentence-temporal category.
Qualitative examples show that while SFT models often get the "vibe" right but the details wrong, the GRPO-refined model recovers specific counts and timestamps accurately.
Limitations and Future Work
- Scope: The model was tested only on laparoscopic cholecystectomy; its ability to generalize to other types of surgery (like robotic or cardiac) remains unproven.
- Linguistic Diversity: The QA pairs were generated via templates. While effective for benchmarking, they don't capture the messy, natural language a clinician might use.
- Atomic Reasoning: The model reasons over 30-second clips. Extending this to reason across an entire multi-hour procedure is a significant future challenge.
Summary updated