SurgQ-VLM: Question-Conditioned Dual-Stream Compression and Answer-Type-Aware Reinforcement for Surgical Video Question Answering

Dhruv Makwana, Ananya MantravadiarXiv 2026 · arXiv

TL;DR

SurgQ-VLM is a new framework for surgical video question answering that tackles the "information bottleneck" of video compression. By using a question-conditioned dual-stream compressor and a reinforcement learning stage with rewards tailored to specific answer types (like counting or timing), it achieves a 10x BLEU-4 gain over zero-shot baselines and significantly outperforms existing surgical video models.

What Problem Does This Paper Address?

Reasoning about surgical video is inherently complex. To document a procedure or assess a surgeon's skill, an AI needs to know not just what is happening, but when it happens, in what order, and for how long.

Current Video Vision-Language Models (VLMs) face two major hurdles in this domain:

  1. Question-Agnostic Compression: Most models compress a 30-second video clip into a fixed number of tokens regardless of the question. If you ask about a split-second instrument change, a generic compressor might have already discarded that specific temporal evidence to save space.
  2. Structural Heterogeneity: Surgical answers aren't just simple labels. They span five incompatible structures: binary (yes/no), classification, multi-label (lists of tools), counting, and sentence-temporal (describing events with timestamps). Standard training signals (like cross-entropy) struggle to score these diverse formats uniformly well.

Methods

The authors propose SurgQ-VLM, a two-stage model based on a LoRA-adapted Qwen2.5-VL-7B.

1. Question-Conditioned Dual-Stream Compression

Instead of treating all video data the same, SurgQ-VLM uses two streams:

  • Slow Stream (0.5 FPS): Captures high-spatial detail for identifying anatomy and phases.
  • Fast Stream (1.0 FPS): Captures rapid instrument transitions and actions.

The "secret sauce" is the GuideInjector. As shown in the pipeline below, visual tokens from both streams are conditioned on the specific question via cross-attention before being fed to the LLM. This allows the model to "look for" the evidence required by the question during the compression process, shrinking 32,805 raw patches down to just 671 highly relevant tokens.

Figure 2. SurgQ-VLM pipeline. Dual-stream sampling (slow: 0.5 FPS/15f; fast: 1.0 FPS/30f) is encoded by a frozen Qwen2.5-VL ViT, then compressed by four question-conditioned GuideInjectors into 671 tokens (405 slow-local + 250 fast-local + 16 global) before a LoRA-adapted LLM. Blue: frozen; green/orange: slow/fast streams; purple: global queries; gold: GuideInjectors. Bottom panel: two-stage training.

2. Answer-Type-Aware Reinforcement Learning

After initial Supervised Fine-Tuning (SFT), the model undergoes a second stage of training using Group Relative Policy Optimization (GRPO). Crucially, the reward function is "answer-type-aware." Rather than a single accuracy score, it dispatches different metrics based on the question's structure:

  • Multi-label: Scored via micro-F1.
  • Counting: Scored by normalized absolute error.
  • Temporal: Scored by a blend of keyword accuracy and timing precision (within ±2 seconds).

Datasets

The authors introduced a massive new multi-task benchmark for surgical video QA:

  • 80,636 QA pairs derived from the well-known Cholec80 and CholecT50 datasets (laparoscopic gallbladder removals).
  • 18 question types covering phases, instruments, and complex "triplets" (instrument-verb-target).
  • Data Split: Partitioned at the video level to ensure no surgery appears in both training and testing, preventing the model from simply memorizing specific procedures.
Table 1. Benchmark distribution across the five answer structures. Q-type inverse-frequency weighting compensates for the imbalance between binary and counting questions.

Key Insights and Ideas

  • Conditioning is the Load-Bearer: The biggest performance drop occurred when question conditioning was removed. Without knowing the question, the compressor loses the specific temporal evidence needed for timing-related answers.
  • Emergent Specialization: By zeroing out different token groups at inference, the authors found that the model naturally learned to use the "Fast" stream for temporal questions and the "Slow" stream for counting and spatial tasks.
  • RL Closes the "Hard Type" Gap: While SFT is good for learning domain vocabulary, it plateaus on structurally difficult questions. The custom GRPO reward provided the necessary gradient signal to improve counting and temporal reasoning where standard cross-entropy failed.

Key Results

SurgQ-VLM sets a new state-of-the-art for surgical VQA:

  • Accuracy: Reached 0.779 keyword accuracy, outperforming specialized models like SurgVidLM (0.740).
  • Language Quality: Achieved a 10x BLEU-4 gain over zero-shot Qwen2.5-VL.
  • GRPO Lift: The reinforcement learning stage added a 2.5 percentage point (pp) boost in accuracy, with the largest gains (+5.1 pp) seen in the difficult sentence-temporal category.

Qualitative examples show that while SFT models often get the "vibe" right but the details wrong, the GRPO-refined model recovers specific counts and timestamps accurately.

Figure 3. Qualitative comparison (sealed test set). Two frames, GT, ZS, SFT, and GRPO per row. Rows 1–3: GRPO recovers on Q03, Q12, and Q05 where ZS and SFT fail. Row 4: all models err on Q13 (failure case).

Limitations and Future Work

  • Scope: The model was tested only on laparoscopic cholecystectomy; its ability to generalize to other types of surgery (like robotic or cardiac) remains unproven.
  • Linguistic Diversity: The QA pairs were generated via templates. While effective for benchmarking, they don't capture the messy, natural language a clinician might use.
  • Atomic Reasoning: The model reasons over 30-second clips. Extending this to reason across an entire multi-hour procedure is a significant future challenge.

Summary updated

Abstract

Reasoning about surgical video -- which phase is underway, which instruments act, and when events occur -- underpins operative documentation, workflow analysis, and skill assessment. Yet surgical visual question answering is still largely built on single frames and closed labels that cannot express a duration, an ordering, or a count, and the video models that lift the frame limit typically compress every clip the same way regardless of the question, discarding the temporal evidence the answer needs. Worse, the answers are structurally heterogeneous, spanning five incompatible structures -- binary, classification, multi-label, counting, and sentence-temporal -- that no single training signal scores uniformly well. We propose SurgQ-VLM, a two-stage model for surgical video question answering. Stage 1 trains a question-conditioned dual-stream (slow/fast) compressor for a LoRA-adapted Qwen2.5-VL-7B: cross-attention conditions both streams on the question, and each clip's visual tokens shrink 49x from the raw ViT patches (11x vs. post-merge backbone tokens). Stage 2 post-trains with GRPO under an answer-type-aware reward that scores each question on the metric matching its answer structure (micro-F1 for label sets, normalised error for counts, a content-timing blend for temporal windows). On a sealed 21,840-question test split over Cholec80 and CholecT50, SurgQ-VLM reaches 0.779 keyword accuracy and a 10x BLEU-4 gain over zero-shot; GRPO adds 2.5 pp K-Acc (2.0 pp attributable to the answer-type-aware reward), concentrated in the structurally hard types, with answer format unchanged (ROUGE-L). Code is available at https://github.com/DhruvMakwana/SurgQ-VLM.
BibTeX
@misc{makwana2026surgq,
  author = {Dhruv Makwana and Ananya Mantravadi},
  title = {{SurgQ-VLM: Question-Conditioned Dual-Stream Compression and Answer-Type-Aware Reinforcement for Surgical Video Question Answering}},
  year = {2026},
  eprint = {2611.01913},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01913},
}
Figure