Abstract
Surgical action triplets describe instrument-tissue interactions as (instrument, verb, target) combinations, supporting detailed analysis of surgical scene activities and workflow. This work focuses on surgical action triplet detection, which is more precise than traditional triplet recognition as it requires joint localization of surgical instruments and recognition of the associated action triplet. Triplet detection is complex due to the lack of spatial triplet annotations. We analyze how instrument spatial annotations affect detection and observe that accurate localization does not guarantee better triplet detection due to the risk of erroneous associations. To address this, we propose MCIT-IG, a two-stage network consisting of a Multi-Class Instrument-aware Transformer and an Interaction Graph. The MCIT stage models per-class target embeddings to reduce misassociation risks, while the IG stage constructs a bipartite dynamic graph to model interactions. Using a mixed-supervised learning strategy with weak target labels and pseudo-triplet labels, our model topped the CholecTriplet challenge at MICCAI 2022.
Introduction
In the pursuit of AI-assisted surgery, achieving "context awareness" is a critical milestone. While many existing systems focus on coarse tasks like surgical phase recognition, true assistance requires a fine-grained understanding of instrument-tissue interactions. These interactions are formalized as surgical action triplets: <instrument, verb, target> (e.g., grasper, dissects, gallbladder).
This research shifts the focus from triplet recognition (identifying if an action occurs in a frame) to triplet detection (identifying which specific instrument is performing what action on which tissue). Detection is significantly more challenging due to two primary factors:
- Data Scarcity: Manually labeling bounding boxes for every interaction across thousands of video frames is labor-intensive and often unavailable.
- Association Ambiguity: Even with precise instrument localization, a frame may contain multiple identical tools performing different actions. Simply finding the tool does not inherently reveal its specific verb or target association.
Methodology: The MCIT-IG Framework
The authors propose a two-stage pipeline named MCIT-IG (Multi-Class Instrument-aware Transformer - Interaction Graph) to solve the association problem under limited supervision.
1. Instrument Detection
The pipeline begins with a standard Deformable DETR model to localize instruments and generate bounding boxes. This provides the spatial foundation for the subsequent stages.
2. Stage 1: Multi-Class Instrument-aware Transformer (MCIT)
The MCIT stage learns "instrument-aware" embeddings for targets (tissues). Rather than processing the entire image globally, the transformer utilizes detected instrument locations to focus its learning. This stage is trained using weak supervision, requiring only frame-level knowledge of target presence rather than precise spatial coordinates.
3. Stage 2: Interaction Graph (IG)
The model constructs a bipartite dynamic graph where one set of nodes represents detected instruments and the other represents the target embeddings from the MCIT stage. The "edges" connecting these nodes represent the verbs. This graph-based approach allows the model to handle multiple instrument instances simultaneously, overcoming a major limitation of traditional frame-level models.
4. Mixed-Supervised Learning
To train the network without explicit spatial triplet labels, the authors employ a mixed-supervision strategy. This involves generating "pseudo-labels" by assigning known frame-level triplets to detected instruments of the matching class. This allows the Interaction Graph to learn associations even in the absence of perfect ground-truth spatial maps.
Experimental Results
The model was evaluated on the CholecT50 dataset, which contains 50 laparoscopic cholecystectomy videos with 100 triplet classes. A subset (Cholec80) was used to pre-train the instrument detector.
Performance Highlights
- State-of-the-Art: MCIT-IG topped the leaderboard of the CholecTriplet challenge at MICCAI 2022.
- Detection Accuracy: The model achieved a triplet detection mean Average Precision (mAP) of 7.32%, a notable improvement over the Rendezvous (RDV) baseline of 6.43%.
- Ablation Insights: The "Tool Awareness" feature—providing the target learner with instrument locations—was the most critical component, boosting triplet detection mAP from 4.98% to 7.32%.
- Real-Time Potential: Despite the two-stage architecture, the model operates at 25 FPS on an Nvidia V100, suggesting viability for near-real-time surgical applications.
Conclusion
The MCIT-IG framework demonstrates that localization alone is insufficient for surgical understanding; the key lies in the semantic association between instruments and their targets. By leveraging target embeddings and bipartite graphs, the model effectively manages the complexities of multi-tool environments. Future work will focus on incorporating temporal information to better distinguish between visually similar actions over time and refining the pseudo-labeling process to reduce training noise.
Summary updated