Using Masked Image Modelling Transformer Architecture for Laparoscopic Surgical Tool Classification and Localization

Hisham ElMoaqet, Rami Janini, Mutaz Ryalat, Ghaith Al-refai, Tamer Abdulbaki Alshirbaji, Nour Aldeen Jalal, Thomas Neumuth, Knut Möeller, Nassir NavabSensors 2025 · DOI

TL;DR

This paper introduces a novel deep learning approach for identifying and locating surgical tools in laparoscopic surgery videos. By leveraging a masked image modeling transformer (BEiT) for classification and its attention mechanisms for weakly supervised localization, the method significantly reduces the need for extensive manual annotations, achieving state-of-the-art results on the Cholec80 benchmark.

What Problem Does This Paper Address?

Minimally Invasive Surgery (MIS) is increasingly common, and AI-powered assistance systems can greatly enhance surgical workflows. A crucial component of these systems is the ability to accurately detect and classify surgical tools within the video feed. However, existing methods face challenges due to complex surgical scenes (occlusion, motion blur, smoke) and a significant bottleneck in data annotation. Manually labeling every tool's location in every frame is time-consuming and requires expert knowledge. This paper aims to overcome these limitations by developing an AI system that can classify and localize surgical tools with reduced annotation requirements.

Methods

The core of this work lies in a two-pronged approach:

  • Surgical Tool Classification: The authors fine-tune the Bidirectional Encoder Representation from Image Transformers (BEiT) model. BEiT is a vision transformer pre-trained using a masked image modeling (MIM) task. This self-supervised pre-training allows BEiT to learn robust visual representations from images without explicit labels. The fine-tuning is performed on the Cholec80 dataset for the specific task of surgical tool classification.
  • Surgical Tool Localization: Instead of requiring precise bounding box annotations, the paper proposes a weakly supervised localization method. This approach leverages the internal attention maps generated by the fine-tuned BEiT model. Specifically, they use Gradient-weighted Class Activation Mapping (Grad-CAM) to visualize the model's attention. These heatmaps are then processed through thresholding, contour detection, and non-maximum suppression to generate bounding boxes for the tools. This method effectively uses only the presence of a tool (binary label) for training, eliminating the need for explicit localization annotations.

To address class imbalance (some tools appear much more frequently than others), the authors incorporate class weights into the loss function during training.

Datasets

  • ImageNet-21k: Used for the initial pre-training of the BEiT model, providing a broad foundation of visual features.
  • Cholec80: A benchmark dataset containing 80 cholecystectomy surgery videos. It is annotated for the presence of seven surgical tools. Videos are downsampled to 1 fps.
  • Cholec80-Boxes: A subset of the Cholec80 dataset containing bounding box labels for surgical tools in five videos, used specifically for evaluating the localization performance.

Key Insights and Ideas

  • Weak Supervision for Localization is Viable: The most significant insight is demonstrating that effective surgical tool localization can be achieved using only binary presence labels, bypassing the need for laborious bounding box annotations. This dramatically reduces the annotation burden.
  • BEiT's Attention is Powerful for Localization: The study highlights that the attention mechanisms within transformer architectures like BEiT, even when not explicitly trained for localization, can implicitly capture spatial information crucial for identifying tool locations.
  • Masked Image Modeling Pre-training Benefits Downstream Tasks: The success of BEiT, pre-trained via masked image modeling, underscores the effectiveness of self-supervised learning for learning generalizable visual features that transfer well to specialized tasks like surgical tool recognition.
  • Class Weights are Crucial for Imbalanced Datasets: The paper explicitly shows the importance of class weighting to ensure that the model learns effectively from underrepresented surgical tools, leading to better overall performance.

Key Results

  • Classification: The BEiT model achieved high performance in surgical tool classification, outperforming previous works that utilized both spatial and temporal information. On the Cholec80 dataset, with 3-fold cross-validation, the average recall, F1 score, and mAP were 91.64%, 93.46%, and 96.43%, respectively. Using a 40/40 split, the mean recall, F1 score, and average precision were 94.21%, 94.91%, and 96.94%.
  • Localization: The weakly supervised localization approach achieved state-of-the-art results. On the Cholec80-Boxes dataset, the proposed model achieved a mean F1 score of 67.6%, which is notable given the minimal supervision. For specific tools like scissors, the F1 score reached 58.6%.
  • Comparison: The BEiT model demonstrated strong performance compared to other existing methods, particularly in classification. While a CNN-based method (CNN_SE_MF) showed slightly better localization performance (mean F1 of 70.1% vs. 67.6%), the BEiT model's strength lies in its classification accuracy and its weakly supervised localization approach.

Limitations and Future Work

  • Annotation Discrepancy: The authors note that the bounding box annotations in the Cholec80-Boxes dataset focus on tool tips, whereas the BEiT model tends to attend to broader regions. This discrepancy can affect Intersection over Union (IoU) scores for localization.
  • Localization vs. Classification Strength: While BEiT excels at classification, a dedicated CNN architecture (CNN_SE_MF) showed slightly better localization performance, suggesting that explicit design for localization can be advantageous.
  • Future Work:
    • Automatic Annotation: The authors plan to develop their own model to automatically annotate the Cholec80 dataset using a Regional-Based Convolutional Neural Network (R-CNN) for more comprehensive evaluation.
  • Spatial-Temporal Model: They aim to implement a spatial-temporal model that leverages information from previous frames to further enhance accuracy and reliability in dynamic surgical environments.

Summary updated

Abstract

Artificial intelligence (AI) has shown its potential to advance applications in various medical fields. One such area involves developing integrated AI-based systems to assist in laparoscopic surgery. Surgical tool detection and phase recognition are key components to develop such systems, and therefore, they have been extensively studied in recent years. Despite significant advancements in this field, previous image-based methods still face many challenges that limit their performance due to complex surgical scenes and limited annotated data. This study proposes a novel deep learning approach for classifying and localizing surgical tools in laparoscopic surgeries. The proposed approach uses a self-supervised learning algorithm for surgical tool classification followed by a weakly supervised algorithm for surgical tool localization, eliminating the need for explicit localization annotation. In particular, we leverage the Bidirectional Encoder Representation from Image Transformers (BEiT) model for tool classification and then utilize the heat maps generated from the multi-headed attention layers in the BEiT model for the localizing of these tools. Furthermore, the model incorporates class weights to address the class imbalance issue resulting from different usage frequencies of surgical tools in surgeries. Evaluated on the Cholec80 benchmark dataset, the proposed approach demonstrated high performance in surgical tool classification, surpassing previous works that utilize both spatial and temporal information. Additionally, the proposed weakly supervised learning approach achieved state-of-the-art results for the localization task.
BibTeX
@inproceedings{elmoaqet2025using,
  author = {Hisham ElMoaqet and Rami Janini and Mutaz Ryalat and Ghaith Al-refai and Tamer Abdulbaki Alshirbaji and Nour Aldeen Jalal and Thomas Neumuth and Knut Möeller and Nassir Navab},
  title = {{Using Masked Image Modelling Transformer Architecture for Laparoscopic Surgical Tool Classification and Localization}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25103017},
}
Figure