TL;DR
SurgStream is a novel framework for real-time surgical video understanding that anchors AI memory and response timing to the structured workflow of an operation. By "folding" detailed instrument-action observations into compact phase summaries and training on specific "clinical windows," the model achieves state-of-the-art performance in proactive assistance while significantly reducing computational overhead.
What Problem Does This Paper Address?
In the high-stakes environment of an operating room, an AI assistant is only useful if it is both accurate and timely. Current streaming video Large Language Models (LLMs) struggle with two major issues in surgery:
- Memory Bloat: Surgical videos are long and visually repetitive. Storing every detail as free-text captions quickly overwhelms an LLM’s context window, making it hard to retrieve specific past events (e.g., "Which was clipped first: the artery or the duct?").
- Timing Uncertainty: Most models are trained to answer questions when asked. However, a proactive assistant needs to know when to speak without being prompted—for instance, reminding a surgeon to prepare a specific tool before the next phase begins. Existing methods often lack explicit supervision for these "clinical deadlines."
SurgStream tackles these by treating the surgical workflow not just as something to recognize, but as the fundamental architecture for the model's memory and decision-making.
Methods
The authors introduce two core components integrated into a streaming LLM (based on Qwen3-VL-4B):
Workflow-Anchored Memory (WAM)
Instead of a messy "stream of consciousness" memory, WAM creates a structured log.
- Observe Records: The model records stable changes in "instrument–action–target" triplets (e.g.,
[12:09] clipper | clip | cystic duct). - Phase Folding: Once a surgical phase ends, the model "folds" all detailed observations from that phase into a single, compact Summary record. This keeps the memory queryable for retrospective questions while preventing the context window from exploding.
Deadline-aware Transition Learning (DTL)
To handle proactivity, the authors define clinical windows for every response. A window has an "earliest admissible time" (when the evidence appears) and a "deadline" (when the response loses its value).
- During training, DTL resamples the target response time within this window in every epoch.
- By favoring earlier times within the window, the model learns to be "eager" but accurate, ensuring it alerts the surgical team before a procedural deadline passes.
Datasets
The researchers utilized two standard surgical datasets, Cholec80 and CholecT50, which contain laparoscopic cholecystectomy (gallbladder removal) videos.
They introduced a new benchmark, SurgStream-Bench, which converts these annotations into 11.7K streaming interactions. The benchmark evaluates models across five temporal modes:
- RT (Real-time perception): What is happening right now?
- BM (Backward memory): Recalling past events and counts.
- FP (Forward proactive QA): Answering questions about future events.
- PA (Proactive alert): Issuing warnings/reminders without a prompt.
- CM (Caption memory): Generating operative notes and summaries.
Key Insights and Ideas
- Workflow as a Compression Algorithm: The most striking insight is that the surgical workflow itself is the best way to compress video data. By "folding" memory at phase boundaries, the model retains the "semantic essence" of the surgery without the "token tax" of raw video frames or redundant captions.
- Supervising Silence: The model doesn't just output text; it outputs control states like
Silence,Standby, andResponse. This allows the model to "think" in the background and only speak when a clinical window opens. - Resampling for Proactivity: By jittering the target response time during training, the authors prevent the model from over-fitting to a single "golden" timestamp, making it more robust to the variable pacing of real surgery.
Key Results
SurgStream was compared against several state-of-the-art models, including MedVLM and OneStreamer.
- Superior Accuracy: SurgStream outperformed fine-tuned competitors across nearly all metrics. Notably, it saw a +12.26 point gain in Forward Proactive QA and a +10.06 point gain in Caption Memory compared to the previous state-of-the-art (OneStreamer).
- Efficiency Gains: Because of the WAM "folding" mechanism, SurgStream reduced peak context length by 46.9% and GPU memory usage by 10.2%.
- Real-Time Performance: The mean update time was 0.53s, well below the 1-second-per-frame input interval required for real-time operation.
Limitations and Future Work
- Procedural Scope: The current evaluation is limited to laparoscopic cholecystectomy. The authors acknowledge that future work must test how well these workflow-anchored concepts generalize to other, more complex surgical procedures.
- Timing Precision: While DTL improves response timing, there is still room to refine the "eagerness" of the model to ensure it doesn't interrupt surgeons at inappropriate moments.
- LLM Judge: The current benchmark uses exact vocabulary matching to avoid the biases of an LLM judge, but this might penalize semantically correct but linguistically different answers.
Overall, SurgStream represents a significant step toward AI that doesn't just watch a surgery, but understands the "rhythm" of the workflow well enough to act as a true digital assistant.
Summary updated