TL;DR
MemSurg is a new framework that gives Vision-Language Models (VLMs) a "structured memory" to better understand surgical videos. By storing and retrieving past observations—like instrument movements and tissue interactions—it allows models to reason about the current scene using the full context of the procedure, leading to significant performance gains in both scene interpretation and workflow recognition.
What Problem Does This Paper Address?
In robot-assisted surgery, a single image frame rarely tells the whole story. An instrument might look the same whether it’s just starting a cut or finishing one, and anatomical structures can deform or be occluded. To truly understand what is happening, a system needs to know what happened before.
Most current methods are "frame-centric," meaning they look at one image at a time or a very short window of frames. This leaves them "blind" to the procedural history and event order (see Figure 1). While some models use temporal windows, they often miss long-term dependencies, such as when an instrument reappears after being hidden for several minutes.
Methods
The authors introduce MemSurg, which shifts the paradigm from "looking" to "remembering then reasoning." The core of the framework is a structured surgical memory graph.
- Memory Node Construction: For every second of video, MemSurg creates a node. This node isn't just a raw image; it contains:
- Episodic Memory ($O_t$): Localized data on entities (instruments, targets) including their categories, bounding boxes, and motion scores derived from optical flow.
- Semantic Relations ($R_t$): A text summary generated by a VLM describing the interactions in that frame (e.g., "Prograsp Forceps grasping Target A").
- Graph Connectivity: Nodes are linked in two ways:
- Chronological edges: Connecting consecutive frames to preserve local order.
- Logical edges: Connecting recurring entities over time, allowing the model to "jump back" to the last time a specific tool was used.
- Task-Adaptive Retrieval: When a question is asked, MemSurg doesn't dump the whole memory. It retrieves a specific subgraph based on the task:
- Frame-wise tasks (e.g., "What is this tool doing?") retrieve recent chronological history.
- Workflow tasks (e.g., "What stage are we in?") retrieve the history of the specific entities currently visible.
- Evidence-Grounded Reasoning: The retrieved history is serialized into text and fed into a reasoning VLM (like GPT-4o or Qwen) alongside the current frame.
Datasets
The framework was evaluated across three diverse surgical benchmarks:
- GraSP: Focused on robot-assisted radical prostatectomy, providing scene and workflow annotations.
- PitVis-2023: Covers endoscopic pituitary surgery with instrument and step labels.
- CholecT50: A laparoscopic cholecystectomy dataset featuring complex instrument-verb-target triplets.
Key Insights and Ideas
- Structured Memory vs. Raw Video: Instead of trying to process massive amounts of video data at once, MemSurg extracts semantic "breadcrumbs." This makes the history searchable and interpretable for the reasoning model.
- Motion as a Semantic Cue: By including motion scores (via RAFT), the model can distinguish between an instrument that is actively engaged in a task and one that is simply idling in the field of view.
- Task-Specific History: The authors found that "more context is not automatically more useful." Their task-adaptive retrieval (choosing between chronological vs. entity-based history) outperformed simply providing both, suggesting that filtering history is as important as recording it.
- Model Agnostic: MemSurg works as a "plug-in." It improved the performance of off-the-shelf models like EndoChat without any additional training.
Key Results
MemSurg demonstrated substantial improvements across the board:
- Vs. Baselines: Achieved a 10.43 percentage-point gain in accuracy over SurgRAW when using GPT-4o.
- Supervised Fine-Tuning: When fine-tuning a Qwen3-VL-8B model, adding memory conditioning improved results by 6.52 points.
- Inference-Time Plug-in: Adding MemSurg to the pretrained EndoChat model increased its accuracy by 7.04 points without any retraining.
- Qualitative Impact: As shown in Figure 3, the memory allows the model to reject "visually plausible" but contextually incorrect answers by checking the preceding interactions.
Limitations and Future Work
The authors identify several areas for growth:
- Offline Normalization: Currently, motion normalization is performed offline (retrospectively), which would need to be adapted for real-time deployment.
- Scalability: As surgeries get longer, workflow histories can grow very large. Future work will look into "bounded retrieval" to keep the context window manageable.
- Rationale Faithfulness: While the model's accuracy improved, ensuring the VLM's internal reasoning perfectly matches the provided evidence remains an ongoing challenge in the field of explainable AI.
Summary updated