Thoracic Surgery Video Analysis for Surgical Phase Recognition

Syed Abdul Mateen, Niharika Malvia, Syed Abdul Khader, Danny Wang, Deepti Srinivasan, Chi-Fu Jeffrey Yang, Lana Schumacher, Sandeep ManjannaarXiv 2024 · DOI · arXiv

Abstract

Automating surgical phase recognition (SPR) is a critical step toward enhancing surgical training, quality assessment, and real-time intraoperative guidance. This paper evaluates Transformer-based architectures for SPR in thoracic surgery, specifically comparing frame-based and video-based models. By utilizing Masked Video Distillation (MVD), we achieved a Top-1 accuracy of 72.93%, outperforming traditional image-based models by over 20%. Our findings emphasize the vital role of temporal context in accurately identifying surgical workflows.

Introduction

In modern operating rooms, robotic-assisted thoracic surgery generates vast amounts of video data. Manually reviewing these hours of footage for educational or quality control purposes is labor-intensive and inefficient. Surgical Phase Recognition (SPR) aims to automate this process by training artificial intelligence to identify specific procedural steps in real-time. This technology is particularly crucial in thoracic surgery, where high-stakes complications may necessitate a rapid transition from minimally invasive techniques to open thoracotomy. Automated analysis can provide surgeons with cognitive guidance, offer objective skill assessments for trainees, and streamline hospital workflows.

Methods

We compared three distinct Transformer-based architectures to determine their efficacy in handling surgical video complexities:

  1. ImageNet ViT (Vision Transformer): A baseline model that analyzes individual frames as static images, lacking temporal awareness.
  2. VideoMAE (Video Masked Autoencoder): A video-specific model that learns motion and temporal continuity by reconstructing masked segments of video sequences.
  3. Masked Video Distillation (MVD): An advanced approach that employs a teacher-student framework to transfer robust features from pre-trained image and video models to a specialized student model.
Overview of the Transformer-based architecture for surgical video analysis.

As shown in the architecture diagram, the video-based models utilize 3D convolutions with a kernel size of [2, 16, 16] to process 16-frame clips, capturing both spatial and temporal information. All models were built on a ViT-L backbone and fine-tuned for 100 epochs.

Datasets

The study utilized a specialized dataset of 17 thoracic surgery videos from Massachusetts General Hospital (MGH), totaling approximately 37 hours of footage. The researchers identified 11 distinct surgical phases, such as "Preparation," "Fissures," and "Disassemble." Videos were processed into 10-second clips using a sliding window method. To ensure the model's ability to generalize to new patients, a strict split was maintained: 13 cases for training and validation, and 4 entirely unseen cases for final testing.

Key Results and Insights

The experimental results highlight a significant performance gap between image-based and video-based architectures.

Quantitative performance comparison of the evaluated models.

The MVD model achieved a Top-1 accuracy of 72.93%, surpassing the frame-based ViT by 20.62 percentage points. This underscores the principle that temporal context is essential for surgical task recognition; while a single frame may appear ambiguous, the motion and progression of tools provide the necessary cues for accurate phase identification. Furthermore, the MVD model's Top-5 accuracy reached 94.14%, indicating that even when the primary prediction was incorrect, the true phase was almost always among the top candidates.

Discussion and Future Work

While the results are promising, the study's reliance on a relatively small dataset of 17 videos suggests that further scaling is needed for production-level reliability. Future research should investigate the model's generalizability across different types of surgical procedures. The ultimate goal is to transition from recognition to active application, such as generating automated surgical summaries or providing real-time alerts to enhance patient safety during complex procedures.

Summary updated

BibTeX
@misc{mateen2024thoracic,
  author = {Syed Abdul Mateen and Niharika Malvia and Syed Abdul Khader and Danny Wang and Deepti Srinivasan and Chi-Fu Jeffrey Yang and Lana Schumacher and Sandeep Manjanna},
  title = {{Thoracic Surgery Video Analysis for Surgical Phase Recognition}},
  year = {2024},
  eprint = {2406.09185},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.09185},
}
Figure