Remembering to Reason: Structured Memory for Scene and Workflow Understanding in Robot-Assisted Surgery

Zhiran Zhang, Hu Wang, Yichen Wu, Yutong XiearXiv 2026 · arXiv

TL;DR

MemSurg is a new framework that gives Vision-Language Models (VLMs) a "structured memory" to better understand surgical videos. By storing and retrieving past observations—like instrument movements and tissue interactions—it allows models to reason about the current scene using the full context of the procedure, leading to significant performance gains in both scene interpretation and workflow recognition.

What Problem Does This Paper Address?

In robot-assisted surgery, a single image frame rarely tells the whole story. An instrument might look the same whether it’s just starting a cut or finishing one, and anatomical structures can deform or be occluded. To truly understand what is happening, a system needs to know what happened before.

Most current methods are "frame-centric," meaning they look at one image at a time or a very short window of frames. This leaves them "blind" to the procedural history and event order (see Figure 1). While some models use temporal windows, they often miss long-term dependencies, such as when an instrument reappears after being hidden for several minutes.

Challenges in frame-centric understanding: missing past interactions and event order makes reasoning difficult.

Methods

The authors introduce MemSurg, which shifts the paradigm from "looking" to "remembering then reasoning." The core of the framework is a structured surgical memory graph.

  1. Memory Node Construction: For every second of video, MemSurg creates a node. This node isn't just a raw image; it contains:
  2. Episodic Memory ($O_t$): Localized data on entities (instruments, targets) including their categories, bounding boxes, and motion scores derived from optical flow.
  3. Semantic Relations ($R_t$): A text summary generated by a VLM describing the interactions in that frame (e.g., "Prograsp Forceps grasping Target A").
  4. Graph Connectivity: Nodes are linked in two ways:
  5. Chronological edges: Connecting consecutive frames to preserve local order.
  6. Logical edges: Connecting recurring entities over time, allowing the model to "jump back" to the last time a specific tool was used.
  7. Task-Adaptive Retrieval: When a question is asked, MemSurg doesn't dump the whole memory. It retrieves a specific subgraph based on the task:
  8. Frame-wise tasks (e.g., "What is this tool doing?") retrieve recent chronological history.
  9. Workflow tasks (e.g., "What stage are we in?") retrieve the history of the specific entities currently visible.
  10. Evidence-Grounded Reasoning: The retrieved history is serialized into text and fed into a reasoning VLM (like GPT-4o or Qwen) alongside the current frame.
Overview of the MemSurg framework, showing node construction and task-adaptive retrieval.

Datasets

The framework was evaluated across three diverse surgical benchmarks:

  • GraSP: Focused on robot-assisted radical prostatectomy, providing scene and workflow annotations.
  • PitVis-2023: Covers endoscopic pituitary surgery with instrument and step labels.
  • CholecT50: A laparoscopic cholecystectomy dataset featuring complex instrument-verb-target triplets.

Key Insights and Ideas

  • Structured Memory vs. Raw Video: Instead of trying to process massive amounts of video data at once, MemSurg extracts semantic "breadcrumbs." This makes the history searchable and interpretable for the reasoning model.
  • Motion as a Semantic Cue: By including motion scores (via RAFT), the model can distinguish between an instrument that is actively engaged in a task and one that is simply idling in the field of view.
  • Task-Specific History: The authors found that "more context is not automatically more useful." Their task-adaptive retrieval (choosing between chronological vs. entity-based history) outperformed simply providing both, suggesting that filtering history is as important as recording it.
  • Model Agnostic: MemSurg works as a "plug-in." It improved the performance of off-the-shelf models like EndoChat without any additional training.

Key Results

MemSurg demonstrated substantial improvements across the board:

  • Vs. Baselines: Achieved a 10.43 percentage-point gain in accuracy over SurgRAW when using GPT-4o.
  • Supervised Fine-Tuning: When fine-tuning a Qwen3-VL-8B model, adding memory conditioning improved results by 6.52 points.
  • Inference-Time Plug-in: Adding MemSurg to the pretrained EndoChat model increased its accuracy by 7.04 points without any retraining.
  • Qualitative Impact: As shown in Figure 3, the memory allows the model to reject "visually plausible" but contextually incorrect answers by checking the preceding interactions.
Example of memory-guided reasoning: the model uses historical records to correctly identify a "pull" action that looks visually similar to "cutting."

Limitations and Future Work

The authors identify several areas for growth:

  • Offline Normalization: Currently, motion normalization is performed offline (retrospectively), which would need to be adapted for real-time deployment.
  • Scalability: As surgeries get longer, workflow histories can grow very large. Future work will look into "bounded retrieval" to keep the context window manageable.
  • Rationale Faithfulness: While the model's accuracy improved, ensuring the VLM's internal reasoning perfectly matches the provided evidence remains an ongoing challenge in the field of explainable AI.

Summary updated

Abstract

Surgical video understanding is an important foundation for intelligent assistance in robot-assisted surgery. Similar surgical views can reflect different actions or procedural stages. Interpreting these views requires access to earlier observations, yet frame-centric prompting leaves this history unavailable. We introduce MemSurg, a framework that remembers surgical observations before reasoning over them. Its structured memory graph stores timestamped entity categories, locations, motion cues, and interaction summaries. Chronological links preserve recent context, while entity-associated links connect recurring observations across the procedure. Task-adaptive retrieval then supplies evidence-grounded prompts for scene and workflow reasoning, supporting both frozen inference and memory-conditioned supervised fine-tuning. Evaluation on robot-assisted prostatectomy and two complementary surgical benchmarks shows a 10.43-percentage-point gain over SurgRAW with GPT-4o. Memory conditioning adds 6.52 points under matched Qwen3-VL-8B fine-tuning, and an inference-time plug-in adds 7.04 points to EndoChat. These results support structured memory as a reusable source of procedural evidence for surgical video understanding. Our code will be publicly released.
BibTeX
@misc{zhang2026remembering,
  author = {Zhiran Zhang and Hu Wang and Yichen Wu and Yutong Xie},
  title = {{Remembering to Reason: Structured Memory for Scene and Workflow Understanding in Robot-Assisted Surgery}},
  year = {2026},
  eprint = {2611.10917},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10917},
}
Figure