Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation

Hongqiu Wang, Yueming Jin, Lei ZhuICRA 2023 · DOI · OpenReview

Abstract

In robot-assisted minimally invasive surgery (RAMIS), accurate surgical reports are essential for clinical documentation, post-operative analysis, and follow-up care. However, generating these reports automatically is challenging due to the complex, dynamic interactions between surgical instruments and biological tissues. Existing deep learning methods often fail to capture these nuances, leading to inaccurate descriptions. This paper introduces a novel neural network architecture designed to boost report generation by explicitly modeling interactive relations through scene graph learning. By integrating temporal node tracking and attention mechanisms, our method significantly outperforms state-of-the-art benchmarks on the 2018 MICCAI Robotic Instrument Segmentation dataset, achieving improvements of 7.48% in BLEU-1 and 36.6% in CIDEr scores.

Introduction

Robot-Assisted Minimally Invasive Surgery (RAMIS) has revolutionized the operating room, yet the administrative burden of documenting these procedures remains high. Manual report generation is labor-intensive and prone to variability. While computer vision has made significant strides in general image captioning, surgical scenes present unique hurdles: instruments frequently move in and out of the field of view, and the most critical information lies not just in the presence of objects, but in the specific actions (e.g., suturing, clipping, or manipulating) occurring between them.

To address these challenges, we propose a framework that moves beyond grid-based image analysis toward a relational understanding of the surgical environment. By treating the scene as a dynamic graph, our model captures the evolving interactions between tools and tissues over time.

Methodology

The proposed framework consists of three primary stages: feature extraction, relational exploration, and interaction perception, culminating in a Transformer-based captioning head.

1. Feature Extraction

The system utilizes YOLOv5 for real-time object detection to identify instruments and tissues. Once regions of interest are localized, a ResNet18 backbone extracts deep visual features for each detected entity, providing the raw data for relational modeling.

2. Relational Exploration (RE) Module

At the core of our approach is a Graph Neural Network (GNN) where nodes represent detected objects and edges represent their potential interactions. The RE module updates node embeddings through message passing, allowing the model to learn context—for instance, how the proximity of a kidney alters the representation of a prograsp forceps. To prevent the "dilution" of unique object features during graph updates, we implement a node reservation operation that preserves original visual features alongside the new relational data.

3. Interaction Perception (IP) Module

The IP module serves as the "control center" for the graph, addressing three specific surgical vision problems:

  • Node Tracking: To handle temporal discontinuity (where tools are momentarily obscured), the module tracks nodes across the current and two previous frames. If an object disappears briefly, its previous state is appended to the graph to maintain context.
  • Global Attention: This acts as a binary switch to determine if any interaction is occurring. If the scene is idle, it suppresses relational learning to prevent the model from "hallucinating" actions.
  • Local Attention: This assigns weights to specific nodes, ensuring the model focuses on the active instruments while ignoring those that are currently stationary or irrelevant to the primary task.

Experimental Results

We validated our model using the 2018 MICCAI Robotic Instrument Segmentation Endoscopy Vision Challenge dataset, which includes 15 robotic nephrectomy videos featuring 11 types of interactive relationships and 9 object classes.

Quantitative Performance

Our method established new state-of-the-art benchmarks across all standard Natural Language Processing (NLP) metrics:

  • BLEU-1: 0.6994 (a 7.48% improvement over the previous leader, CIDA).
  • ROUGE: 0.7038 (a 5.43% improvement).
  • CIDEr: 3.9006 (a massive 36.6% improvement).

The significant jump in the CIDEr metric is particularly noteworthy, as CIDEr is designed to measure how well generated text matches human consensus, suggesting our reports are much more aligned with how surgeons actually describe procedures.

Qualitative Analysis

Qualitatively, the model demonstrates a superior ability to distinguish between active and idle states. While baseline models often struggle with "idle tool" errors—describing a stationary forceps as active—our Interaction Perception module effectively filters these false positives, correctly identifying when a tool is merely present versus when it is "manipulating tissue."

Conclusion and Future Work

By explicitly modeling the dynamic relationships between instruments and tissues through scene graph learning, our approach provides a more accurate and robust solution for automated surgical reporting. While the current model relies solely on visual data, future iterations could benefit from incorporating robotic kinematic data to further refine action recognition. Additionally, expanding the dataset to include a wider variety of surgical procedures will be essential for generalizing these results across the field of robotic surgery.

Summary updated

Abstract

For robot-assisted surgery, an accurate surgical report reflects clinical operations during surgery and helps document entry tasks, post-operative analysis and follow-up treatment. It is a challenging task due to many complex and diverse interactions between instruments and tissues in the surgical scene. Although existing surgical report generation methods based on deep learning have achieved large success, they often ignore the interactive relation between tissues and instrumental tools, thereby degrading the report generation performance. This paper presents a neural network to boost surgical report generation by explicitly exploring the interactive relation between tissues and surgical instruments. To do so, we first devise a relational exploration (RE) module to model the interactive relation via graph learning, and an interaction perception (IP) module to assist the graph learning in RE module. In our IP module, we first devise a node tracking system to identify and append missing graph nodes of the current video frame for constructing graphs at RE module. Moreover, the IP module generates a global attention model to indicate the existence of the interactive relation on the whole scene of the current video frame to eliminate the graph learning at the current video frame. Furthermore, our IP module predicts a local attention model to more accurately identify the interaction relation of each graph node for assisting the graph updating at the RE module. After that, we concatenate features of all graph nodes of RE module and pass concatenated features into a transformer for generating the output surgical report. We validate the effectiveness of our method on a widely-used robotic surgery benchmark dataset, and experimental results show that our network can significantly outperform existing state-of-the-art surgical report generation methods (e.g., 7.48% and 5.43% higher for BLEU-1 and ROUGE).
BibTeX
@inproceedings{wang2023dynamic,
  author = {Hongqiu Wang and Yueming Jin and Lei Zhu},
  title = {{Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160647},
}
Figure