Surgical Instruction Generation with Transformers

Jinglu Zhang, Yinyu Nie, Jian Chang, Jian-Jun Zhang

Introduction

Intra-operative guidance is critical for surgical safety, particularly during rare procedures or in environments where expert mentoring is unavailable. While telementoring provides a partial solution, it is often hindered by high costs, bandwidth limitations, and legal complexities. Previous AI research in surgical assistance has largely focused on phase recognition or gesture classification—tasks that label existing actions but fail to generate complex, context-aware instructions.

To address this, researchers have developed a framework for "Surgical Instruction Generation," where a model must jointly interpret visual activity in a surgical scene and translate it into coherent natural language. Moving beyond traditional Recurrent Neural Networks (RNNs), which struggle with long-range dependencies and vanishing gradients, this work leverages the Transformer architecture combined with reinforcement learning to provide high-fidelity surgical guidance.

Methodology

The proposed framework utilizes a Transformer-backboned encoder-decoder network designed to capture non-local relationships within surgical images and generate precise textual instructions.

Model Architecture

  1. Visual Feature Extraction: Spatial features are extracted from surgical images using a pre-trained ResNet-101 backbone.
  2. Transformer Encoder: A stack of six attention blocks processes visual features, allowing the model to understand the spatial relationships between different regions of the surgical field.
  3. Transformer Decoder: A corresponding six-block stack generates instructions word-by-word. It employs masked self-attention to maintain temporal consistency and encoder-decoder attention to align the generated text with specific visual features.
The Transformer-based encoder-decoder architecture for surgical instruction generation.

Self-Critical Reinforcement Learning

Standard models trained with "Teacher Forcing" often suffer from exposure bias, where small errors during testing compound over time. To mitigate this, the authors implemented Self-Critical Reinforcement Learning (RL). This approach directly optimizes the CIDEr score—a metric designed to measure human-like captioning—as a reward function, ensuring the model's output aligns more closely with expert surgical descriptions.

Experimental Setup and Dataset

The model was evaluated using the Database for AI Surgical Instruction (DAISI), which contains 17,255 color images from 290 diverse medical procedures, including laparoscopic surgery and fetal monitoring. After cleaning the dataset to remove noisy image-text pairs, the final study utilized 16,413 images, split into training (13,094), validation (1,646), and testing (1,673) sets.

Key Results and Discussion

The Transformer + RL model demonstrated a significant performance leap over previous Bi-RNN baselines across all primary metrics:

  • Fluency (BLEU-4): Improved from 9.3% to 44.9%.
  • Human-likeness (CIDEr): Increased from 8.32 to 42.7.

Qualitative analysis confirms the model's ability to handle complex multimodal contexts, successfully generating detailed instructions for intricate tasks such as mesh fixation and peritoneal flap closure.

Qualitative results comparing model predictions against ground truth (GT) instructions across various surgical stages.

Ablation studies highlighted that while the Transformer architecture itself outperformed LSTMs, the addition of Reinforcement Learning was the catalyst for the most substantial performance gains, suggesting that optimizing for human-centric metrics leads to more robust task representations.

Limitations and Future Directions

Despite these advancements, several challenges remain. The DAISI dataset is relatively small compared to general-domain datasets like COCO, increasing the risk of overfitting. Furthermore, the current model processes the image as a whole; incorporating fine-grained supervision through object detection (e.g., identifying specific tools or anatomical structures) could improve accuracy. Finally, the "one caption constraint" in current evaluation metrics may unfairly penalize valid alternative descriptions of the same surgical step. Future work will focus on developing specialized medical pre-trained models and expanding the diversity of reference captions.

Summary updated

BibTeX
@inproceedings{zhang2021surgical,
  author = {Jinglu Zhang and Yinyu Nie and Jian Chang and Jian-Jun Zhang},
  title = {{Surgical Instruction Generation with Transformers}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=LlXjAnacwHJ},
}
Figure