EndoSCOP-V: A Multi-Turn Video Understanding Evaluation Framework for Multimodal Models in Endoscopy Reporting and Clinical Reasoning

Abhishek Tyagi, Tanmay Jain, Vilas Biradar, Krithi Krishna Koduri, Misba Hul Bashir Masoodi, Swetha Rapolu, Hardik Rughwani, Rachana Kotha, Anjan Kumar Kaipa, Anudeep Katrevula, Rakesh Garlapati, Aniruddha Pratap Singh, Rajendra Patel, Abhay Tyagi, Sundeep Lakhtakia, Mohan Ramchandani, Rakesh Kalapala, D. Nageshwar ReddyarXiv 2026 · arXiv

TL;DR

EndoSCOP-V is a new video-grounded, multi-turn evaluation framework designed to test multimodal models on the complex, hierarchical reasoning required for real-world endoscopy reporting. By moving beyond static images to long-horizon video and conditional clinical dialogues, it reveals a significant 23-percentage-point performance gap between state-of-the-art models and board-certified endoscopists.

What Problem Does This Paper Address?

In a real clinical setting, an endoscopist doesn't just look at a single frame and name a disease. Diagnosis is a dynamic process: a finding in a video triggers a "chain of reasoning" that spans several minutes. As shown in Figure 1, identifying an ulcer leads to immediate follow-up questions: Where exactly is it? What is its morphology? How severe is it? Should we biopsy or intervene?

Figure 1: Endoscopy reporting is inherently multi-turn. A single finding cascades through a chain of follow-up questions (for example, Disease? →Ulcer, Organ? →Duodenum, Sub-location? → Duodenal Bulb, Wall? →Posterior, Grade? →IIB) that mirrors how an endoscopist composes a structured EHR report. EndoSCOP-V evaluates models on this multi-turn dialogue rather than on isolated single-shot recognition. Illustration generated with Gemini 3.1 Pro.

Existing benchmarks for Large Multimodal Models (LMMs) in endoscopy are poorly matched to this reality. Most rely on static images or short, single-disease clips and evaluate models using "one-shot" questions. This misses the temporal context of video and the conditional nature of clinical reporting—where the answer to one question (e.g., "Is there an ulcer?") fundamentally dictates the relevance of the next (e.g., "How big is it?").

Methods

The authors introduce several novel methodological choices to create a more "deployment-realistic" benchmark:

  • Markovian Context-Pruned Oracle-Forcing: To evaluate long dialogues without letting a single early mistake ruin the entire sequence, the framework uses "oracle forcing." Before each turn, the "gold" (correct) answer to the previous question is injected into the model's dialogue history. This isolates the model's reasoning at each specific step, preventing "cascading errors."
  • EHR-Grounded Question Generation: Instead of using LLMs to hallucinate questions, the authors used 1,033 clinician-reviewed templates mapped directly to structured Electronic Health Record (EHR) fields. This ensures the questions mirror the actual data endoscopists are required to record.
  • Key-Frame Anchors: Clinicians marked specific "anchor" frames for each finding. While models are fed uniformly sampled frames from the full video, these anchors allow researchers to distinguish between a model failing to find a lesion (temporal localization) and failing to understand it (comprehension).
  • NDR Accuracy: The authors highlight that binary "is condition X present?" questions are often "format-trivial" for models, leading to inflated scores. They introduce Non-Disease-Rejection (NDR) accuracy, which excludes these easier items to provide a more rigorous measure of clinical reasoning.

Datasets

The core of the work is the EndoSCOP-V dataset, which significantly raises the bar for endoscopy benchmarks in terms of scale and complexity.

Table 1: Compact comparison of EndoSCOP-V with peer endoscopy benchmarks. Appendix D (Table 16) gives the full version with an eval-items column.
  • Scale: 1,307 real-world EGD and colonoscopy videos totaling 63.2 hours.
  • Quality: 97% of videos are 1080p HD at native acquisition resolution.
  • Breadth: Covers 54 diseases from the World Endoscopy Organization (WEO) MST 3.0 terminology.
  • Complexity: Unlike many benchmarks that focus on single pathologies, 22% of cases in EndoSCOP-V feature co-existing diseases (up to six per case), reflecting real-world clinical "case mix."
  • Ground Truth: 20,274 question-answer turns derived from clinician-filled reports, with a 100-case subset further validated by a five-rater consensus.

Key Insights and Ideas

  • The "Recognize-but-Fail-to-Characterize" Gap: Models are surprisingly good at simple disease identification (DI-M task) but struggle significantly when asked to compose multiple visual cues into categorical outputs, such as multi-variable severity grading (e.g., Mayo or UCEIS scores).
  • Video-Native Tasks: Frame-ablation studies proved that certain tasks—like recognizing a biopsy action (AR-B) or navigating anatomical structures (AR-V)—are "video-native." They cannot be solved even by an "oracle" single frame; they require the temporal information only a video provides.
  • Specialization vs. Generalization: Interestingly, a model specifically fine-tuned for colonoscopy (ColonR1) performed worse on colonoscopy tasks than generalist models like Qwen3.6. This suggests that narrow medical fine-tuning might sometimes sacrifice general visual reasoning capabilities.
  • Scale Isn't Everything: In the Qwen family, the 9B parameter model outperformed the larger 27B and 32B versions on several metrics, indicating that architectural choices or training data mix may be more critical than raw parameter count for this domain.

Key Results

The benchmark establishes a clear hierarchy of performance, with a substantial gap still remaining between AI and human experts.

Table 3: Accuracy (%). Best per column in bold.
  • Clinician Baseline: A panel of 15 board-certified endoscopists achieved 77.9% strict accuracy.
  • Open-Weight Leaderboard: The strongest model, Qwen3.6-27B, reached 59.8% strict overall accuracy, but this dropped to 53.4% on the more challenging NDR metric.
  • Proprietary Models: In a spot-check, Gemini 3-flash was the top performer at 61.4% overall, yet it still trailed the clinician panel by 23 percentage points on strict NDR accuracy.
  • Task Performance: Action Recognition (AR) was the "easiest" task (80.7% best), while Clinical Reasoning (CR) and Grading & Scoring (GS) remained the most difficult (both ~40-46% best).

Limitations and Future Work

The authors acknowledge several limitations:

  • Single-Institution Data: All cases come from one major hospital in India; multi-institution validation is a necessary next step.
  • Structured Output Only: The framework currently evaluates only deterministic, verifiable choices. It does not yet assess free-form narrative generation or the model's ability to express calibrated uncertainty.
  • Assistive vs. Replacement: The 23-point gap suggests current LMMs are suited for assistive roles (e.g., drafting a report for a doctor to verify) rather than autonomous reporting.

Future research could involve testing "agentic" systems—models that can use external tools or memory—on EndoSCOP-V to see if they can close the reasoning gap with human clinicians.

Summary updated

Abstract

Endoscopic diagnosis unfolds over native video, where a single positive finding during traversal of the gastrointestinal tract sets off a multi-minute chain of reasoning over location, morphology, severity and intervention. Existing endoscopy multimodal benchmarks test only static images or single-disease videos, which miss the temporal and clinical reasoning that real-world reporting demands. We introduce EndoSCOP-V, a video-grounded multi-turn evaluation framework for structured endoscopy reporting and clinical reasoning. It contains 1,307 real-world endoscopy and colonoscopy videos (63.2 h; 97% 1080p) averaging about 174 s. Its 20,274 question-answer turns are instantiated deterministically from clinician-filled structured endoscopy-reporting EHRs, so no per-case annotation is synthetic or LLM-rephrased. They come from 1,033 clinician-reviewed templates, about 18 per disease, that span a clinically aligned 9-task x 30-sub-task reporting taxonomy over 54 World Endoscopy Organization (WEO) MST 3.0 diseases. 22% of procedures carry co-existing pathologies, up to six per case. A Markovian context-pruned oracle-forcing protocol evaluates the dialogue turn by turn, injecting the gold answer before each subsequent turn so that per-turn reasoning is isolated from cascading dialogue errors. A panel of 15 board-certified endoscopists sets a 78% strict-accuracy baseline on 1,500 questions. Across 14 open-weight LMMs, the best 27B-class model reaches 59.8% strict accuracy, falling to 53.4% once format-trivial disease-rejection items are excluded. On a between-subjects spot-check, the best of 3 frontier proprietary models trails the clinician panel by 23 pp on strict NDR accuracy. Sub-task analysis reveals consistent gaps in quantitative measurement, multi-variable grading and intervention reasoning on long-horizon video. Data: Hugging Face (EndoSCOP-V). Code: GitHub (abhi25t/EndoSCOP-V).
BibTeX
@misc{tyagi2026endoscop,
  author = {Abhishek Tyagi and Tanmay Jain and Vilas Biradar and Krithi Krishna Koduri and Misba Hul Bashir Masoodi and Swetha Rapolu and Hardik Rughwani and Rachana Kotha and Anjan Kumar Kaipa and Anudeep Katrevula and Rakesh Garlapati and Aniruddha Pratap Singh and Rajendra Patel and Abhay Tyagi and Sundeep Lakhtakia and Mohan Ramchandani and Rakesh Kalapala and D. Nageshwar Reddy},
  title = {{EndoSCOP-V: A Multi-Turn Video Understanding Evaluation Framework for Multimodal Models in Endoscopy Reporting and Clinical Reasoning}},
  year = {2026},
  eprint = {2611.10626},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10626},
}
Figure