SurgStream: A Workflow-Anchored Proactive Streaming Framework and Benchmark for Real-Time Surgical Video Understanding

Ruipeng Ma, Qing Xu, Jie Liang, Zhen ChenarXiv 2026 · arXiv

TL;DR

SurgStream is a novel framework for real-time surgical video understanding that anchors AI memory and response timing to the structured workflow of an operation. By "folding" detailed instrument-action observations into compact phase summaries and training on specific "clinical windows," the model achieves state-of-the-art performance in proactive assistance while significantly reducing computational overhead.

What Problem Does This Paper Address?

In the high-stakes environment of an operating room, an AI assistant is only useful if it is both accurate and timely. Current streaming video Large Language Models (LLMs) struggle with two major issues in surgery:

  1. Memory Bloat: Surgical videos are long and visually repetitive. Storing every detail as free-text captions quickly overwhelms an LLM’s context window, making it hard to retrieve specific past events (e.g., "Which was clipped first: the artery or the duct?").
  2. Timing Uncertainty: Most models are trained to answer questions when asked. However, a proactive assistant needs to know when to speak without being prompted—for instance, reminding a surgeon to prepare a specific tool before the next phase begins. Existing methods often lack explicit supervision for these "clinical deadlines."

SurgStream tackles these by treating the surgical workflow not just as something to recognize, but as the fundamental architecture for the model's memory and decision-making.

Methods

The authors introduce two core components integrated into a streaming LLM (based on Qwen3-VL-4B):

Workflow-Anchored Memory (WAM)

Instead of a messy "stream of consciousness" memory, WAM creates a structured log.

  • Observe Records: The model records stable changes in "instrument–action–target" triplets (e.g., [12:09] clipper | clip | cystic duct).
  • Phase Folding: Once a surgical phase ends, the model "folds" all detailed observations from that phase into a single, compact Summary record. This keeps the memory queryable for retrospective questions while preventing the context window from exploding.
The WAM process: detailed observations are folded into phase-level summaries to keep memory compact.

Deadline-aware Transition Learning (DTL)

To handle proactivity, the authors define clinical windows for every response. A window has an "earliest admissible time" (when the evidence appears) and a "deadline" (when the response loses its value).

  • During training, DTL resamples the target response time within this window in every epoch.
  • By favoring earlier times within the window, the model learns to be "eager" but accurate, ensuring it alerts the surgical team before a procedural deadline passes.
DTL supervises the model to respond within a valid clinical window, favoring early responses.

Datasets

The researchers utilized two standard surgical datasets, Cholec80 and CholecT50, which contain laparoscopic cholecystectomy (gallbladder removal) videos.

They introduced a new benchmark, SurgStream-Bench, which converts these annotations into 11.7K streaming interactions. The benchmark evaluates models across five temporal modes:

  • RT (Real-time perception): What is happening right now?
  • BM (Backward memory): Recalling past events and counts.
  • FP (Forward proactive QA): Answering questions about future events.
  • PA (Proactive alert): Issuing warnings/reminders without a prompt.
  • CM (Caption memory): Generating operative notes and summaries.
Fig. 1. Illustration of SurgStream and SurgStream-Bench. (a) An example of streaming surgical interaction, including current-scene questions, retrospective queries, proactive alerts and memory writing. (b) The five temporal modes evaluated by SurgStream-Bench.

Key Insights and Ideas

  • Workflow as a Compression Algorithm: The most striking insight is that the surgical workflow itself is the best way to compress video data. By "folding" memory at phase boundaries, the model retains the "semantic essence" of the surgery without the "token tax" of raw video frames or redundant captions.
  • Supervising Silence: The model doesn't just output text; it outputs control states like Silence, Standby, and Response. This allows the model to "think" in the background and only speak when a clinical window opens.
  • Resampling for Proactivity: By jittering the target response time during training, the authors prevent the model from over-fitting to a single "golden" timestamp, making it more robust to the variable pacing of real surgery.

Key Results

SurgStream was compared against several state-of-the-art models, including MedVLM and OneStreamer.

  • Superior Accuracy: SurgStream outperformed fine-tuned competitors across nearly all metrics. Notably, it saw a +12.26 point gain in Forward Proactive QA and a +10.06 point gain in Caption Memory compared to the previous state-of-the-art (OneStreamer).
  • Efficiency Gains: Because of the WAM "folding" mechanism, SurgStream reduced peak context length by 46.9% and GPU memory usage by 10.2%.
  • Real-Time Performance: The mean update time was 0.53s, well below the 1-second-per-frame input interval required for real-time operation.

Limitations and Future Work

  • Procedural Scope: The current evaluation is limited to laparoscopic cholecystectomy. The authors acknowledge that future work must test how well these workflow-anchored concepts generalize to other, more complex surgical procedures.
  • Timing Precision: While DTL improves response timing, there is still room to refine the "eagerness" of the model to ensure it doesn't interrupt surgeons at inappropriate moments.
  • LLM Judge: The current benchmark uses exact vocabulary matching to avoid the biases of an LLM judge, but this might penalize semantically correct but linguistically different answers.

Overall, SurgStream represents a significant step toward AI that doesn't just watch a surgery, but understands the "rhythm" of the workflow well enough to act as a true digital assistant.

Summary updated

Abstract

Surgical video understanding is essential for intraoperative assistance such as workflow monitoring, instrument preparation and procedural documentation. Delivering such assistance in real time requires a model to jointly perceive the ongoing scene, remember past surgical events, and respond before clinically relevant moments pass. Existing approaches provide limited support for jointly retaining procedural evidence and deciding when to respond in streaming surgical video. In this work, we propose SurgStream, a workflow-anchored proactive streaming framework for real-time surgical video understanding. We first introduce Workflow-Anchored Memory (WAM) to record structured instrument-action-target observations and fold them into phase-level summaries, providing compact and queryable surgical memory. Deadline-aware Transition Learning (DTL) is then devised to supervise response states within clinical windows, enabling timely responses before procedural deadlines. We further introduce SurgStream-Bench, comprising 11.7K streaming interactions across various tasks covering real-time perception, backward memory queries, forward proactive QA, proactive alerts and caption memory. Experiments on SurgStream-Bench show gains over state-of-the-art methods, especially in memory and forward proactive QA. The source code will be available.
BibTeX
@misc{ma2026surgstream,
  author = {Ruipeng Ma and Qing Xu and Jie Liang and Zhen Chen},
  title = {{SurgStream: A Workflow-Anchored Proactive Streaming Framework and Benchmark for Real-Time Surgical Video Understanding}},
  year = {2026},
  eprint = {2611.01982},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01982},
}
Figure