EndoComparer: Comparative Surgical Video Question Answering with Pseudo-Siamese Visual Adaptation and Perceive-then-Compare Reasoning

Shi Li, Lalith Sharan, Joël L. Lavanchy, Kun Yuan, Lorenzo Arboit, Nabani Banik, Elif Yazici, Maria Vannucci, Vinkle Srivastav, Pietro Mascagni, Nicolas PadoyarXiv 2026 · arXiv

TL;DR

Researchers have developed EndoComparer, a vision-language model (VLM) designed to "spot the difference" between two surgical videos. By using a specialized visual adapter for each video and a "perceive-then-compare" reasoning strategy, the model significantly outperforms general-purpose AI in identifying variations in surgical skill, tool usage, and anatomy.

What Problem Does This Paper Address?

In surgical training, learning often happens through comparison: “Why was the dissection in Video A cleaner than in Video B?” While Vision-Language Models (VLMs) have become adept at describing single videos, they struggle with comparative reasoning.

The core challenge is the "binding problem." When a model looks at two videos of the same procedure (e.g., a gallbladder removal), it often gets confused about which observation belongs to which video. Because the organs, tools, and background look nearly identical across different cases, standard models frequently attribute a specific action to the wrong video slot. This makes them unreliable for providing the nuanced feedback required for surgical education.

Methods

The authors introduce two novel components to help the model keep its "eyes" and "thoughts" organized:

  1. Pseudo-Siamese Visual LoRA (PSVL): Standard models use a "Siamese" approach, applying the exact same weights to both videos. EndoComparer uses a shared, frozen vision encoder but adds a dedicated Low-Rank Adapter (LoRA) for each video slot. This creates separate visual pathways for "Video 1" and "Video 2," helping the model maintain the identity of each stream from the very first layer of processing without doubling the computational cost.
  2. Perceive-then-Compare Chain of Thought (PtC-CoT): Instead of jumping straight to an answer, the model is trained to follow a specific reasoning path. It must first state what is happening in Video 1, then state what is happening in Video 2, and only then derive the comparative answer. This "thinking out loud" ensures the final judgment is grounded in distinct observations for each video.

Datasets

The researchers curated EndoDiffVQA, a new benchmark for comparative surgical VQA.

  • Scale: Nearly 11,000 multiple-choice questions (four options each).
  • Source Material: Derived from 1,762 clips of laparoscopic cholecystectomy (gallbladder removal) and Roux-en-Y gastric bypass.
  • Categories: The questions cover four critical domains: Anatomy (visibility, color, state), Tools (presence, count, engagement), Actions (interactions, goals), and Skill (proficiency scores).
  • Distractors: Unlike random multiple-choice options, the "wrong" answers are designed to mimic common model failures, such as swapping the facts between the two videos.

Key Insights and Ideas

  • Attribution is Harder than Perception: The study found that models often "see" the correct facts but "attach" them to the wrong video. This confirms that the bottleneck in comparative VQA isn't just visual recognition, but the logical binding of facts to sources.
  • The "Video-Swap" Litmus Test: To prove their model wasn't just guessing, the authors performed a swap control—exchanging the order of Video 1 and Video 2. While standard fine-tuned VLMs saw their accuracy plummet (losing 5–8 points), EndoComparer remained robust, losing only 0.6 points. This proves the model truly understands which video is which.
  • Skill is the Ultimate Challenge: Comparing surgical skill is much harder than identifying a tool. The gains from EndoComparer were most pronounced in the "Skill" category, where the model must integrate observations over minutes of footage rather than just identifying a single frame.

Key Results

  • State-of-the-Art Performance: EndoComparer achieved a 54.0 Strict score, outperforming its fine-tuned backbone (Qwen3.5-9B) by 5.7 points.
  • Beating Proprietary Giants: It surpassed Gemini 3.6 Flash by 7.1 points, despite Gemini being a much larger, proprietary model.
  • Synergy: Ablation studies showed that PSVL (the visual fix) and PtC-CoT (the reasoning fix) are complementary; using both together yielded significantly better results than either alone, especially for skill assessment.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Two-Video Limit: The current framework is limited to comparing exactly two videos at a time.
  • Multiple Choice Constraints: The benchmark uses a multiple-choice format, which measures if a model can select the right relation but doesn't test its ability to describe differences in free-form natural language.
  • Noise Inheritance: The dataset is built on existing captions and expert scores, meaning it inherits any noise or inter-rater variability present in those source materials.
  • Clinical Utility: While a strong step toward AI coaching, the model currently acts more as a "triage" tool for ranking cases rather than a source of open-ended, unsupervised surgical feedback.

Summary updated

Abstract

Purpose: Surgical training benefits from explicit comparison between cases: in patient condition, in technique and in skill proficiency. Surgical vision-language models (VLMs) offer a promising and flexible interface for this purpose. Yet current VLMs typically see one video at a time, and they often fall short when asked to compare videos. Methods: We propose EndoComparer, a VLM for comparative surgical video question answering with two components. (a) A Pseudo-Siamese Visual LoRA (PSVL) gives each video slot its own low-rank adapter over a shared frozen vision encoder, separating the two videos in the visual pathway at the cost of a single encoder pass. (b) A Perceive-then-Compare chain of thought (PtC-CoT) trains the model to first state what each video shows, then derive the answer from these statements. We further curate EndoDiffVQA, a benchmark of nearly 11K multiple-choice questions on video-pair over cholecystectomy and gastric bypass procedures, covering anatomy, tool, action and skill. Results: EndoComparer outperforms strong generalist and surgical VLMs on detecting and specifying difference between videos. Ablations show that two proposed components contribute complimentarily, and the model remains more robust under a video-swap control that exposes confusion between the two videos. Conclusion: Providing a dedicated visual reasoning path through PSVL and PtC-CoT makes cross-case comparison of surgical videos more robust for VLMs. EndoDiffVQA and EndoComparer provide a benchmark and a strong baseline for this task. Project page: https://camma-public.github.io/endocomparer
BibTeX
@misc{li2026endocomparer,
  author = {Shi Li and Lalith Sharan and Joël L. Lavanchy and Kun Yuan and Lorenzo Arboit and Nabani Banik and Elif Yazici and Maria Vannucci and Vinkle Srivastav and Pietro Mascagni and Nicolas Padoy},
  title = {{EndoComparer: Comparative Surgical Video Question Answering with Pseudo-Siamese Visual Adaptation and Perceive-then-Compare Reasoning}},
  year = {2026},
  eprint = {2611.19578},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19578},
}
Figure