LoViT: Long Video Transformer for Surgical Phase Recognition

Yang Liu, Maxence Boels, Luis C. García-Peraza-Herrera, Tom Vercauteren, Prokar Dasgupta, Alejandro Granados, Sébastien OurselinarXiv 2023 · DOI · OpenReview

TL;DR

This paper introduces LoViT, a novel Transformer-based framework for surgical phase recognition in long videos. LoViT effectively fuses local and global temporal information using a multi-scale temporal aggregator and a phase transition-aware supervision mechanism, achieving state-of-the-art performance on challenging surgical datasets.

What Problem Does This Paper Address?

Surgical phase recognition is crucial for developing intelligent tools that can monitor surgical workflows, quantify performance, and provide real-time feedback. However, existing methods struggle with long surgical videos due to computational constraints and the difficulty of fusing local and global temporal information. Frame-level supervision for spatial feature extraction can lead to misclassifications when similar frames appear across different phases. Furthermore, traditional temporal models like LSTMs have limitations in capturing long-range dependencies, and even Transformers, while powerful, can be computationally prohibitive for extended video sequences. This paper aims to address these limitations by proposing a method that can efficiently process long surgical videos and accurately recognize phases by effectively integrating multi-scale temporal information and leveraging phase transition cues.

Methods

LoViT employs a two-stage approach. First, it utilizes a temporally-rich spatial feature extractor. Unlike standard methods that train on individual frames, LoViT trains its spatial feature extractor using a set of frames sampled from the beginning of a phase up to the current frame. This allows the extractor to learn more robust spatial representations by considering temporal context, even when computational resources are limited.

Second, a multi-scale temporal feature aggregator processes these spatially encoded features. This aggregator consists of:

  • Cascaded L-Trans modules: These modules capture local, fine-grained temporal information using vanilla self-attention. They process features at different temporal scales (small and large local windows).
  • G-Informer module: This module leverages a ProbSparse self-attention mechanism to efficiently capture global temporal relationships across long sequences, overcoming the quadratic complexity of standard Transformers.
  • Multi-scale temporal fusion head: This head combines the local and global features extracted by L-Trans and G-Informer.

Crucially, LoViT incorporates phase transition-aware supervision. It generates a phase transition map using an asymmetric Gaussian kernel to highlight critical temporal information around phase transitions. This map is used alongside the standard phase classification loss to guide the model's learning, emphasizing the importance of accurately identifying phase changes.

Datasets

The proposed method was evaluated on two publicly available surgical video datasets:

  • Cholec80: Contains 80 laparoscopic cholecystectomy videos, with an average duration of 39 minutes. It has 7 surgical phases.
  • AutoLaparo: Consists of 21 laparoscopic hysterectomy videos, with an average duration of 66 minutes. It also has 7 surgical phases.

Both datasets were downsampled to 1 fps and resized to 250x250 pixels for consistency.

Key Insights and Ideas

  • Temporally-rich spatial features are crucial: Training spatial feature extractors with temporal context, even from a limited set of frames, significantly improves their ability to distinguish between similar-looking frames from different surgical phases.
  • Multi-scale temporal aggregation is key: Combining local (fine-grained actions) and global (overall phase context) temporal information is essential for robust phase recognition in long videos.
  • Phase transitions are informative: Explicitly supervising the model to learn phase transition points, using a phase transition map, significantly enhances its ability to understand the temporal relationships between phases.
  • ProbSparse attention for efficiency: The use of ProbSparse self-attention in the G-Informer module is a critical innovation for handling long video sequences efficiently without sacrificing performance.

Key Results

LoViT consistently outperformed state-of-the-art methods on both Cholec80 and AutoLaparo datasets across various metrics.

  • Cholec80: LoViT achieved a video-level accuracy of 91.50±6.10%, outperforming Trans-SVNet by 2.4 percentage points. It also achieved a phase-level Jaccard index of 74.15%.
  • AutoLaparo: LoViT achieved a video-level accuracy of 81.43±7.35%, outperforming Trans-SVNet by 3.1 percentage points. It showed significant improvements in phase-level metrics, with a 20.9 pp increase in precision and a 5.2 pp increase in Jaccard compared to Trans-SVNet.
  • Qualitative results showed LoViT's ability to better distinguish between similar frames and capture phase transitions more accurately than previous methods.

Limitations and Future Work

  • Computational Cost for Inference: While the ProbSparse attention in G-Informer reduces complexity, the model still requires feeding all previous spatial features into the temporal model for recognition. This can lead to deteriorating inference speed for extremely long surgeries. Future work could explore methods to avoid redundant calculations by utilizing previous analysis results to reduce time and memory costs.
  • Unusual Operation Processes: The paper notes that LoViT, like other methods, still struggles to accurately recognize some phases that appear in unusual operation processes, as seen in Figure 6 where a phase appears out of sequence. Future research could focus on discovering more complex relationships between phases to handle such anomalies.

Summary updated

Abstract

Online surgical phase recognition plays a significant role towards building contextual tools that could quantify performance and oversee the execution of surgical workflows. Current approaches are limited since they train spatial feature extractors using frame-level supervision that could lead to incorrect predictions due to similar frames appearing at different phases, and poorly fuse local and global features due to computational constraints which can affect the analysis of long videos commonly encountered in surgical interventions. In this paper, we present a two-stage method, called Long Video Transformer (LoViT) for fusing short- and long-term temporal information that combines a temporally-rich spatial feature extractor and a multi-scale temporal aggregator consisting of two cascaded L-Trans modules based on self-attention, followed by a G-Informer module based on ProbSparse self-attention for processing global temporal information. The multi-scale temporal head then combines local and global features and classifies surgical phases using phase transition-aware supervision. Our approach outperforms state-of-the-art methods on the Cholec80 and AutoLaparo datasets consistently. Compared to Trans-SVNet, LoViT achieves a 2.4 pp (percentage point) improvement in video-level accuracy on Cholec80 and a 3.1 pp improvement on AutoLaparo. Moreover, it achieves a 5.3 pp improvement in phase-level Jaccard on AutoLaparo and a 1.55 pp improvement on Cholec80. Our results demonstrate the effectiveness of our approach in achieving state-of-the-art performance of surgical phase recognition on two datasets of different surgical procedures and temporal sequencing characteristics whilst introducing mechanisms that cope with long videos.
BibTeX
@misc{liu2023lovit,
  author = {Yang Liu and Maxence Boels and Luis C. García-Peraza-Herrera and Tom Vercauteren and Prokar Dasgupta and Alejandro Granados and Sébastien Ourselin},
  title = {{LoViT: Long Video Transformer for Surgical Phase Recognition}},
  year = {2023},
  eprint = {2305.08989},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.08989},
}
Figure