Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions

Saurav Sharma, Chinedu Innocent Nwoye, Didier Mutter, Nicolas Padoy

Abstract

Surgical action triplets describe instrument-tissue interactions as (instrument, verb, target) combinations, supporting detailed analysis of surgical scene activities and workflow. This work focuses on surgical action triplet detection, which is more precise than traditional triplet recognition as it requires joint localization of surgical instruments and recognition of the associated action triplet. Triplet detection is complex due to the lack of spatial triplet annotations. We analyze how instrument spatial annotations affect detection and observe that accurate localization does not guarantee better triplet detection due to the risk of erroneous associations. To address this, we propose MCIT-IG, a two-stage network consisting of a Multi-Class Instrument-aware Transformer and an Interaction Graph. The MCIT stage models per-class target embeddings to reduce misassociation risks, while the IG stage constructs a bipartite dynamic graph to model interactions. Using a mixed-supervised learning strategy with weak target labels and pseudo-triplet labels, our model topped the CholecTriplet challenge at MICCAI 2022.

Introduction

In the pursuit of AI-assisted surgery, achieving "context awareness" is a critical milestone. While many existing systems focus on coarse tasks like surgical phase recognition, true assistance requires a fine-grained understanding of instrument-tissue interactions. These interactions are formalized as surgical action triplets: <instrument, verb, target> (e.g., grasper, dissects, gallbladder).

This research shifts the focus from triplet recognition (identifying if an action occurs in a frame) to triplet detection (identifying which specific instrument is performing what action on which tissue). Detection is significantly more challenging due to two primary factors:

  1. Data Scarcity: Manually labeling bounding boxes for every interaction across thousands of video frames is labor-intensive and often unavailable.
  2. Association Ambiguity: Even with precise instrument localization, a frame may contain multiple identical tools performing different actions. Simply finding the tool does not inherently reveal its specific verb or target association.

Methodology: The MCIT-IG Framework

The authors propose a two-stage pipeline named MCIT-IG (Multi-Class Instrument-aware Transformer - Interaction Graph) to solve the association problem under limited supervision.

1. Instrument Detection

The pipeline begins with a standard Deformable DETR model to localize instruments and generate bounding boxes. This provides the spatial foundation for the subsequent stages.

2. Stage 1: Multi-Class Instrument-aware Transformer (MCIT)

The MCIT stage learns "instrument-aware" embeddings for targets (tissues). Rather than processing the entire image globally, the transformer utilizes detected instrument locations to focus its learning. This stage is trained using weak supervision, requiring only frame-level knowledge of target presence rather than precise spatial coordinates.

3. Stage 2: Interaction Graph (IG)

The model constructs a bipartite dynamic graph where one set of nodes represents detected instruments and the other represents the target embeddings from the MCIT stage. The "edges" connecting these nodes represent the verbs. This graph-based approach allows the model to handle multiple instrument instances simultaneously, overcoming a major limitation of traditional frame-level models.

4. Mixed-Supervised Learning

To train the network without explicit spatial triplet labels, the authors employ a mixed-supervision strategy. This involves generating "pseudo-labels" by assigning known frame-level triplets to detected instruments of the matching class. This allows the Interaction Graph to learn associations even in the absence of perfect ground-truth spatial maps.

Experimental Results

The model was evaluated on the CholecT50 dataset, which contains 50 laparoscopic cholecystectomy videos with 100 triplet classes. A subset (Cholec80) was used to pre-train the instrument detector.

Performance Highlights

  • State-of-the-Art: MCIT-IG topped the leaderboard of the CholecTriplet challenge at MICCAI 2022.
  • Detection Accuracy: The model achieved a triplet detection mean Average Precision (mAP) of 7.32%, a notable improvement over the Rendezvous (RDV) baseline of 6.43%.
  • Ablation Insights: The "Tool Awareness" feature—providing the target learner with instrument locations—was the most critical component, boosting triplet detection mAP from 4.98% to 7.32%.
  • Real-Time Potential: Despite the two-stage architecture, the model operates at 25 FPS on an Nvidia V100, suggesting viability for near-real-time surgical applications.

Conclusion

The MCIT-IG framework demonstrates that localization alone is insufficient for surgical understanding; the key lies in the semantic association between instruments and their targets. By leveraging target embeddings and bipartite graphs, the model effectively manages the complexities of multi-tool environments. Future work will focus on incorporating temporal information to better distinguish between visually similar actions over time and refining the pseudo-labeling process to reduce training noise.

Summary updated

Abstract

Surgical action triplets describe instrument-tissue interactions as (instrument, verb, target) combinations, thereby supporting a detailed analysis of surgical scene activities and workflow. This work focuses on surgical action triplet detection, which is challenging but more precise than the traditional triplet recognition task as it consists of joint (1) localization of surgical instruments and (2) recognition of the surgical action triplet associated with every localized instrument. Triplet detection is highly complex due to the lack of spatial triplet annotation. We analyze how the amount of instrument spatial annotations affects triplet detection and observe that accurate instrument localization does not guarantee better triplet detection due to the risk of erroneous associations with the verbs and targets. To solve the two tasks, we propose MCIT-IG, a two-stage network, that stands for Multi-Class Instrument-aware Transformer-Interaction Graph. The MCIT stage of our network models per class embedding of the targets as additional features to reduce the risk of misassociating triplets. Furthermore, the IG stage constructs a bipartite dynamic graph to model the interaction between the instruments and targets, cast as the verbs. We utilize a mixed-supervised learning strategy that combines weak target presence labels for MCIT and pseudo triplet labels for IG to train our network. We observed that complementing minimal instrument spatial annotations with target embeddings results in better triplet detection. We evaluate our model on the CholecT50 dataset and show improved performance on both instrument localization and triplet detection, topping the leaderboard of the CholecTriplet challenge in MICCAI 2022.
BibTeX
@inproceedings{sharma2023surgical,
  author = {Saurav Sharma and Chinedu Innocent Nwoye and Didier Mutter and Nicolas Padoy},
  title = {{Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  url = {https://openreview.net/forum?id=sxGDXjPYGz},
}
Figure