SIG-Former: monocular surgical instruction generation with transformers

Jinglu Zhang, Yinyu Nie, Jian ChangInt. J. Comput. Assist. Radiol. Surg 2022 · DOI

Introduction

In modern clinical settings, the demand for intra-operative assistance and automated surgical training tools is rapidly increasing. While telementoring—remote guidance via telecommunications—offers a potential solution, it is often limited by latency, legal complexities, and the scarcity of human experts. To address these challenges, researchers have developed SIG-Former, a transformer-based framework designed to interpret a surgical field from monocular RGB images and generate descriptive, natural language instructions for the next steps.

Surgical Instruction Generation (SIG) is inherently difficult due to the cluttered and variable nature of surgical environments. Translating high-dimensional visual features into coherent text requires bridging a significant "modal gap." Previous approaches primarily relied on Recurrent Neural Networks (RNNs), which often struggle with long-term dependencies and suffer from "exposure bias"—a cumulative error effect during text generation where small mistakes early in a sequence lead to nonsensical outputs.

The SIG-Former Architecture

To overcome the limitations of sequential recurrence, SIG-Former leverages the parallel attention mechanisms of the Transformer architecture. This allows the model to simultaneously attend to different regions of a surgical image while deciding which word to generate next, effectively capturing complex spatial and temporal relationships.

The SIG-Former architecture utilizes a transformer encoder-decoder structure to map visual features from a surgical image to natural language instructions.

The system begins with visual feature extraction using a pre-trained ResNet-101. This produces an "attentive feature map," which is projected into a 512-dimensional space to serve as input for the transformer. The core of the model consists of a 6-layer Encoder and a 6-layer Decoder. The Encoder models spatial relationships within the image, while the Decoder employs masked self-attention for text embeddings and cross-attention to align words with specific visual features.

A critical innovation in SIG-Former is the use of Self-Critical Reinforcement Learning (SCST). After initial training with standard cross-entropy loss, the model is fine-tuned to directly maximize the CIDEr score—a metric that evaluates how well generated captions align with human consensus. This reinforcement learning stage helps mitigate exposure bias by using the model's own test-time inference as a baseline to normalize rewards, penalizing low-quality "greedy" predictions.

Dataset and Preprocessing

The model was rigorously evaluated using the DAISI (Database for AI Surgical Instruction) dataset. This database contains over 17,000 color images from 290 clinical procedures across 20 different medical disciplines, including laparoscopic hernia repair and gastrectomy.

Analysis of the DAISI dataset showing the distribution of common words and the length of surgical instruction sequences.

The authors performed extensive preprocessing, cleaning the dataset to 16,413 high-quality image-caption pairs and normalizing text (e.g., expanding medical abbreviations). As shown in the dataset analysis, the instructions often involve specific anatomical terms and action verbs. The sequence length distribution indicates that while many instructions are concise (6-15 words), a significant portion requires more complex descriptions. An intra-procedural split was used for training and testing to ensure the model learned the specific "flow" of surgical procedures.

Key Results and Insights

SIG-Former established a new state-of-the-art on the DAISI dataset, significantly outperforming previous benchmarks:

  • Quantitative Performance: On the intra-procedure split, the model achieved a CIDEr score of 45.61 and a BLEU-4 of 48.2. In comparison, the original Bi-RNN baseline achieved a CIDEr score of only 8.32.
  • Ablation Study: Adding the SCST reinforcement learning stage improved the CIDEr score by over 4 points (from 41.32 to 45.61), proving that direct metric optimization is essential for high-quality text generation.
  • Qualitative Accuracy: Visualizations demonstrated that SIG-Former correctly identifies specific anatomical landmarks, such as the "hernia sac" or "fascial closure," whereas simpler models often produced generic or repetitive phrases.

Limitations and Future Work

Despite these successes, several hurdles remain. The DAISI dataset is relatively small compared to general computer vision datasets like COCO, which can lead to overfitting. Furthermore, the current reliance on a single reference caption per image does not account for the multiple valid ways a surgical step can be described. Future research will likely focus on developing larger, multi-reference datasets and exploring domain-specific pre-training on medical imagery to further enhance the model's clinical utility.

Summary updated

BibTeX
@article{zhang2022sig,
  author = {Jinglu Zhang and Yinyu Nie and Jian Chang and Jian-Jun Zhang},
  title = {{SIG-Former: monocular surgical instruction generation with transformers}},
  journal = {Int. J. Comput. Assist. Radiol. Surg.},
  year = {2022},
  doi = {10.1007/s11548-022-02718-9},
}
Figure