GRAC: Graph-Based Retrieval-Augmented Captioning for Surgical Scene Description

Rochak Dhakal, Chen Zhao, Soup Cobb, Xin Qian, Xiaofeng Yang, Weihua ZhouarXiv 2026 · arXiv

TL;DR

GRAC (Graph-Based Retrieval-Augmented Captioning) is a novel framework that improves automated surgical scene description by combining structured scene graphs with Retrieval-Augmented Generation (RAG). By retrieving similar past surgical activities and explicitly comparing them to the current scene, GRAC significantly boosts both the linguistic quality and factual accuracy of generated captions.

What Problem Does This Paper Address?

Generating automated surgical reports is a high-stakes task. These reports are essential for training, performance review, and documentation, but creating them manually is a massive time sink for clinicians.

Current AI approaches usually try to map visual features directly to natural language. However, this is difficult because annotated surgical datasets are often small, and models frequently struggle to maintain "factual agreement"—for example, they might correctly identify a tool but hallucinate the wrong action or anatomical target. Since surgical procedures are highly repetitive and follow structured patterns, the authors argue that models should leverage "prior knowledge" from similar past cases to guide the generation of new descriptions.

Methods

The authors introduce GRAC, which moves away from simple image-to-text mapping in favor of a four-stage pipeline:

  1. Edge-Aware Scene Graph Encoding: Instead of just looking at pixels, GRAC represents a surgical frame as a graph where nodes are instruments/tissues and edges are their interactions. It uses a GATv2 (Graph Attention Network) to create a rich, context-aware representation of the scene.
  2. Caption-Supervised Retrieval: Unlike standard RAG that might retrieve based on visual similarity, GRAC is trained to retrieve prior graph-caption pairs based on surgical activity. It learns an embedding space where scenes with similar instrument-action-target triplets are clustered together.
  3. Graph Comparison: This is a key innovation. GRAC doesn't just blindly use the retrieved captions; it compares the retrieved scene graph to the current query graph. It looks for agreement and discrepancies (e.g., "The retrieved case used a Prograsp forceps, but my current case is using a Bipolar forceps").
  4. Conditioned Generation: A Transformer decoder generates the final caption by attending to both the current scene features and the "memory" of the retrieved, comparison-conditioned captions.
Figure 1: Overview of GRAC. The input surgical scene graph is encoded to capture surgical in- teractions and retrieve similar prior scene graphs with their corresponding captions. GRAC then compares the retrieved graphs with the current scene and uses these comparisons to guide caption generation through a Transformer decoder.

Datasets

The model was evaluated on two primary benchmarks:

  • EndoVis-18: A widely used robotic nephrectomy (kidney surgery) dataset. The authors used a subset of 392 frames for evaluation and over 1,100 for training.
  • TORS (Transoral Robotic Surgery): A smaller dataset used to test if the model's performance gains could be replicated in a different surgical setting.

Key Insights and Ideas

  • Retrieval as a Semantic Prior: The paper demonstrates that surgical descriptions are repetitive enough that "retrieval" acts as a powerful shortcut. Even a simple baseline that just "copies" the caption of the most similar training example performs surprisingly well.
  • Activity over Appearance: Retrieving based on the structure of the activity (who is doing what to what) is more effective for captioning than retrieving based on how the image looks.
  • The Power of Comparison: Simply adding retrieved text can sometimes confuse a model. By explicitly calculating a "comparison vector" between the old and new graphs, GRAC can "correct" the retrieved information to fit the current reality.
  • Reducing Hallucinations: One of the most significant "aha" moments in the results is that GRAC significantly reduces "instrument hallucination"—the error of naming a tool that isn't actually in the frame.

Key Results

GRAC set new performance benchmarks on the EndoVis-18 dataset:

  • Linguistic Quality: It achieved a CIDEr-D score of 5.88, a massive jump over the previous state-of-the-art (SGT) at 4.23 and standard Transformers at 3.36.
  • Factual Accuracy: In terms of "triplet-set accuracy" (getting the instrument, action, and target all correct), GRAC reached 52.6%, compared to just 32.7% for SGT.
  • Generalization: On the TORS dataset, GRAC maintained its lead with a CIDEr-D of 5.06, while the strongest baseline trailed at 3.80.

Limitations and Future Work

While GRAC is a significant step forward, the authors note several areas for improvement:

  • Frame-level vs. Sequence-level: Currently, the model describes individual frames. Real operative reports need to summarize entire sequences of events over time.
  • Action Correction: While the model is great at identifying the right instruments, it still sometimes struggles to "correct" the action if the retrieved example has the right tools but a different movement (e.g., "retracting" vs. "cutting").
  • Dataset Size: The study relies on relatively small, procedure-specific datasets. Testing on more diverse surgical procedures is necessary to prove clinical reliability.
  • Automated Graphs: This study used ground-truth scene graphs. Future work will need to integrate automated scene graph generation from raw video to create a truly end-to-end system.

Summary updated

Abstract

Purpose: Surgical operative reports support procedural documentation, surgical training, performance assessment, and workflow review. However, extracting meaningful information often requires extensive manual review and clinical expertise. Automated surgical scene descriptions could reduce this burden by leveraging recurring surgical activities and prior knowledge. Methods: We propose GRAC (Graph-Based Retrieval-Augmented Captioning), a framework that represents surgical scenes as graphs capturing relationships among instruments, actions, and anatomical structures. Given an input scene graph, GRAC learns to retrieve relevant graph-caption pairs from training data. It compares instrument-tissue interactions between retrieved examples and the input scene, using comparison features to condition retrieved-caption representations during generation. Results: On the EndoVis-18 benchmark, GRAC achieved a CIDEr-D of 5.88, compared with 4.23 for SGT and 3.36 for the strongest conventional Transformer baseline. Exact instrument-action-target triplet-set accuracy reached 52.6%, compared with 32.7% for SGT and 26.2% for the strongest conventional baseline. These results demonstrate improvements in language quality and factual agreement. On TORS, GRAC achieved a CIDEr-D of 5.06, compared with 3.80 for the strongest baseline. Conclusion: GRAC demonstrates that combining structured surgical scene representations with retrieved examples improves surgical scene captioning and produces more contextually meaningful descriptions.
BibTeX
@misc{dhakal2026grac,
  author = {Rochak Dhakal and Chen Zhao and Soup Cobb and Xin Qian and Xiaofeng Yang and Weihua Zhou},
  title = {{GRAC: Graph-Based Retrieval-Augmented Captioning for Surgical Scene Description}},
  year = {2026},
  eprint = {2611.09866},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09866},
}
Figure