SurgRAW: Multi-Agent Workflow with Chain-of-Thought Reasoning for Surgical Intelligence

Chang Han Low, Ziyue Wang, Tianyi Zhang, Zhitao Zeng, Zhu Zhuo, Evangelos B. Mazomenos, Yueming Jin

TL;DR

SurgRAW is a novel multi-agent AI framework that transforms robotic surgical video analysis from a "black-box" classification task into a transparent, reasoning-driven process. By organizing specialized AI agents into a hierarchical workflow—featuring "panel discussions" for conflict resolution and medical knowledge retrieval—it outperforms existing supervised models by over 14% in accuracy while providing clear, step-by-step clinical explanations.

The Challenges of Surgical AI

In the high-stakes environment of Robotic-Assisted Surgery (RAS), scene understanding requires more than simple tool identification; it demands an understanding of procedural logic and future state prediction. Current surgical AI systems face three primary hurdles:

  1. Fragmentation: Most models are designed for isolated tasks (e.g., instrument detection), failing to achieve a unified understanding of the surgical scene.
  2. The "Black Box" Problem: Existing systems often provide answers without interpretable justifications, hindering trust in safety-critical medical settings.
  3. The Domain Gap: General Vision-Language Models (VLMs) like GPT-4o frequently hallucinate or lack the specific anatomical and procedural knowledge required for nuanced surgical tasks.

Methodology: The SurgRAW Framework

SurgRAW addresses these issues by mimicking the collaborative nature of a surgical team. The architecture utilizes a hierarchical structure to manage complexity and ensure clinical accuracy.

The SurgRAW architecture features an Orchestrator, specialized CoT-embedded agents, and a Panel Discussion mechanism for quality control.

Core Components:

  • The Orchestrator: This top-level agent classifies incoming queries into two streams: Visual-Semantic (VS) for perceptual tasks (e.g., "What is this tool?") and Cognitive-Inference (CI) for logical tasks (e.g., "What is the next step?").
  • Specialized Agents: Dedicated agents, such as the Instrument Specialist and Action Predictor, utilize custom Chain-of-Thought (CoT) prompts. These clinically aligned sequences guide the AI to localize features, extract relevant data, and verify findings against procedural norms.
  • Panel Discussion (VS Tasks): To mitigate hallucinations, an "Action Evaluator" moderates a debate between agents. It cross-references logic against a Surgical Knowledge Graph to ensure, for instance, that a predicted action is physically possible with the identified instrument.
  • Retrieval-Augmented Generation (RAG) (CI Tasks): For complex logical inference, the system queries a library of medical resources (e.g., MedlinePlus) to ground predictions in established clinical guidelines.

SurgCoTBench: A Reasoning-Centric Dataset

To evaluate this framework, the researchers developed SurgCoTBench, a dataset comprising 14,256 vision-language QA pairs across 2,277 curated frames from radical prostatectomy and lobectomy procedures. Unlike traditional datasets, SurgCoTBench includes frame-level annotations specifically designed to evaluate the reasoning process rather than just the final output.

Experimental Results

SurgRAW demonstrates significant improvements over both specialized supervised models and general-purpose VLMs. Using a Qwen2-VL backbone, SurgRAW achieved 64.03% overall accuracy, outperforming the specialized Surgical-VQA model by 14.61%.

Comparison of reasoning chains between standard GPT-4o and SurgRAW, illustrating how structured CoT prompts lead to more accurate surgical identification.

Key performance highlights include:

  • Zero-Shot Superiority: SurgRAW significantly outperformed mainstream VLMs like GPT-4o and LLaVA-Med without requiring task-specific fine-tuning.
  • Deep Inference Gains: In "Patient Detail" extraction, SurgRAW reached 96.21% accuracy, a massive leap from the 9.44% achieved by standard GPT-4o.
  • Reliability: The system maintained a low performance deviation (±1.51%), indicating much higher stability than supervised baselines (±5.46%).

Discussion and Future Directions

While SurgRAW sets a new benchmark for surgical intelligence, challenges remain. The current framework operates on individual frames, meaning it does not yet fully capture the continuous temporal flow of a complete surgical video. Additionally, the multi-agent nature of the workflow introduces latency that may impact real-time intraoperative use.

Future work will focus on incorporating temporal dynamics, expanding SurgCoTBench to a wider variety of surgical procedures, and integrating "surgeon-in-the-loop" feedback to further refine the system's clinical reasoning chains.

Summary updated

BibTeX
@article{low2026surgraw,
  author = {Chang Han Low and Ziyue Wang and Tianyi Zhang and Zhitao Zeng and Zhu Zhuo and Evangelos B. Mazomenos and Yueming Jin},
  title = {{SurgRAW: Multi-Agent Workflow with Chain-of-Thought Reasoning for Surgical Intelligence}},
  journal = {IEEE Robotics and Automation Letters},
  year = {2026},
  doi = {10.1109/lra.2026.3665443},
}
Figure