Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions

Saurav Sharma, Chinedu Innocent Nwoye, Didier Mutter, Nicolas Padoy

TL;DR

Researchers have developed MCIT-IG, a two-stage deep learning framework that doesn't just recognize what is happening in a surgery, but where it’s happening. By combining a transformer-based target embedding stage with a dynamic interaction graph, the model achieves state-of-the-art results in surgical action triplet detection (identifying the instrument, the action, and the tissue involved) even when spatial data is scarce.

What Problem Does This Paper Address?

In the world of AI-assisted surgery, "context awareness" is the holy grail. Most existing systems perform coarse tasks like identifying the phase of a surgery. However, to truly assist a surgeon, AI needs to understand fine-grained interactions, known as surgical action triplets: <instrument, verb, target> (e.g., grasper, dissects, gallbladder).

The challenge is moving from triplet recognition (is this happening in the frame?) to triplet detection (which specific instrument is doing what to which tissue?). Detection is significantly harder because:

  1. Data Scarcity: Manually labeling the exact bounding box for every instrument-tissue interaction in thousands of video frames is incredibly tedious.
  2. Association Ambiguity: Even if you know where the instruments are, a single frame might contain multiple identical tools (e.g., two graspers) doing different things. Simply localizing the tool doesn't tell you which verb or target it belongs to.

Methods

The authors propose a two-stage pipeline called MCIT-IG (Multi-Class Instrument-aware Transformer - Interaction Graph):

  1. Instrument Detection: They use a standard Deformable DETR model to find the instruments and draw bounding boxes.
  2. Stage 1: MCIT: This stage learns "instrument-aware" embeddings for the targets (tissues). Instead of just looking at the whole image, the transformer uses the detected instrument locations to focus its learning. It is trained using weak supervision (only knowing if a target is present in the frame, not where).
  3. Stage 2: IG (Interaction Graph): The model builds a bipartite graph where one set of nodes represents detected instruments and the other represents the target embeddings from Stage 1. The "edges" (connections) between these nodes represent the verbs.
  4. Mixed Supervision: To train the graph without explicit spatial triplet labels, the authors generate "pseudo-labels." They take the known frame-level triplets and assign them to detected instruments that match the class, allowing the model to learn associations without perfect ground-truth maps.

Datasets

  • CholecT50: The primary dataset for training and evaluation, containing 50 laparoscopic cholecystectomy videos labeled with 100 triplet classes.
  • Cholec80: A subset of this larger dataset (12 videos) was used specifically to train the instrument detector with spatial bounding box annotations.

Key Insights and Ideas

  • Localization is not enough: The authors found that simply having a better instrument detector doesn't automatically lead to better triplet detection. If the model can't correctly associate the tool with the tissue, the extra precision is wasted.
  • Embeddings as "Soft" Labels: By learning per-class embeddings for targets (Stage 1), the model creates a rich feature set that captures the "semantics" of the scene. This helps the Interaction Graph (Stage 2) make much smarter guesses about which tool is interacting with which tissue.
  • The Power of Graphs: Casting the problem as a dynamic bipartite graph allows the model to handle multiple instrument instances simultaneously, a major hurdle for older, frame-level recognition models.

Key Results

  • Leaderboard Success: MCIT-IG topped the leaderboard of the CholecTriplet challenge at MICCAI 2022.
  • Detection Performance: The model achieved a triplet detection mAP (mean Average Precision) of 7.32%, a significant jump over the Rendezvous (RDV) baseline of 6.43%.
  • Efficiency: Despite its complexity, the model runs at 25 FPS on an Nvidia V100, making it potentially viable for near-real-time surgical assistance.
  • Ablation Success: The "Tool Awareness" feature (letting the target learner know where the tools are) was crucial, boosting triplet detection mAP from 4.98% to 7.32%.

Limitations and Future Work

  • Instrument Detector Dependency: The system's performance is heavily capped by the accuracy of the initial instrument detector. If the detector misses a tool, the entire triplet for that tool is lost.
  • Pseudo-label Noise: The mixed supervision strategy relies on pseudo-labels, which can be "noisy" (incorrectly assigned) during training.
  • Future Directions: The authors suggest that incorporating temporal information (looking at sequences of frames rather than just one) could further improve the model's ability to distinguish between similar-looking actions.

Summary updated

Abstract

Surgical action triplets describe instrument-tissue interactions as 〈instrument, verb, target〉 combinations, thereby supporting a detailed analysis of surgical scene activities and workflow. This work focuses on surgical action triplet detection, which is challenging but more precise than the traditional triplet recognition task as it consists of joint (1) localization of surgical instruments and (2) recognition of the surgical action triplet associated with every localized instrument. Triplet detection is highly complex due to the lack of spatial triplet annotation. We analyze how the amount of instrument spatial annotations affects triplet detection and observe that accurate instrument localization does not guarantee a better triplet detection due to the risk of erroneous associations with the verbs and targets. To solve the two tasks, we propose MCIT-IG, a two-stage network, that stands for M ulti-C lass I nstrument-aware T ransformer -I nteraction G raph. The MCIT stage of our network models per class embedding of the targets as additional features to reduce the risk of misassociating triplets. Furthermore, the IG stage constructs a bipartite dynamic graph to model the interaction between the instruments and targets, cast as the verbs. We utilize a mixed-supervised learning strategy that combines weak target presence labels for MCIT and pseudo triplet labels for IG to train our network. We observed that complementing minimal instrument spatial annotations with target embeddings results in better triplet detection. We evaluate our model on the CholecT50 dataset and show improved performance on both instrument localization and triplet detection, topping the leaderboard of the CholecTriplet challenge in MICCAI 2022.
BibTeX
@inproceedings{sharma2023surgical,
  author = {Saurav Sharma and Chinedu Innocent Nwoye and Didier Mutter and Nicolas Padoy},
  title = {{Surgical Action Triplet Detection by Mixed Supervised Learning of Instrument-Tissue Interactions}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  url = {https://openreview.net/forum?id=sxGDXjPYGz},
}

Loading related content...

Figure