OperA: Attention-Regularized Transformers for Surgical Phase Recognition

Tobias Czempiel, Magdalini Paschali, Daniel Ostler, Seong Tae Kim, Benjamin Busam, Nassir Navab

Abstract

In this paper we introduce OperA, a transformer-based model that accurately predicts surgical phases from long video sequences. A novel attention regularization loss encourages the model to focus on high-quality frames during training. Moreover, the attention weights are utilized to identify characteristic high attention frames for each surgical phase, which could further be used for surgery summarization. OperA is thoroughly evaluated on two datasets of laparoscopic cholecystectomy videos, outperforming various state-of-the-art temporal refinement approaches.

Introduction

Surgical Phase Recognition is a critical component for the "Operating Room of the Future," enabling AI systems to provide real-time guidance, safety alerts, and automated procedure summarization. However, laparoscopic video data is inherently challenging due to its length and visual "noise"—including camera movement, smoke from cauterization, and blood obscuring the field of view.

Traditional sequential models like LSTMs often struggle with long-range dependencies in 30-60 minute procedures, while Temporal Convolutional Networks (TCNs) have limited receptive fields. OperA (Attention-Regularized Transformers) addresses these challenges by leveraging the global context of Transformers while introducing a novel mechanism to filter out uninformative visual data.

Methodology

OperA utilizes a transformer-based architecture designed for online sequential modeling. The pipeline consists of three main stages:

  1. Feature Extraction: A ResNet-50 backbone, pre-trained on surgical tasks, extracts visual features from each frame.
  2. Causal Transformer Stack: The core architecture comprises 11 self-attention layers. To enable real-time prediction, the model employs causal masking, ensuring that predictions at any time $t$ only depend on current and past frames.
  3. Attention Regularization ($L_{reg}$): This novel loss function penalizes the model for assigning high attention weights to frames where the CNN backbone exhibits high error or low confidence. By tying attention to the backbone's performance, the model learns to prioritize high-quality, informative frames over noisy ones.
The OperA architecture featuring a CNN backbone and a causal Transformer stack with attention regularization.

Experimental Setup and Results

The model was evaluated on two laparoscopic cholecystectomy datasets: Cholec80 (80 videos, 7 phases) and MitiSW (85 videos, 8 phases). OperA was compared against state-of-the-art baselines including TeCNO and ResLSTM.

Quantitative performance comparison on Cholec80 and MitiSW datasets alongside qualitative prediction timelines.

As shown in the results, OperA achieved a state-of-the-art accuracy of 91.26% on Cholec80, a significant improvement over previous temporal refinement methods. Ablation studies confirmed that the attention regularization term consistently improved both accuracy and F1-scores across both datasets.

Discussion: Attention as a Filter

A key insight from this work is the dual utility of the attention mechanism. Beyond improving classification accuracy, the attention weights serve as a natural filter for video quality. Frames with the Highest Attention (HA) typically contain clear, characteristic views of surgical actions (e.g., clipping a vessel), while Lowest Attention (LA) frames often correspond to obstructed views or motion blur. This property allows OperA to be used for automatic surgical summarization by selecting "highlight" frames based on attention scores.

Conclusion and Future Work

OperA demonstrates that attention-regularized transformers are highly effective for modeling long-term dependencies in surgical workflows while remaining robust to visual noise. Future work will explore scaling the model beyond 11 layers—currently limited by hardware constraints—and testing its generalization across a wider variety of laparoscopic procedures.

Summary updated

Abstract

In this paper we introduce OperA, a transformer-based model that accurately predicts surgical phases from long video sequences. A novel attention regularization loss encourages the model to focus on high-quality frames during training. Moreover, the attention weights are utilized to identify characteristic high attention frames for each surgical phase, which could further be used for surgery summarization. OperA is thoroughly evaluated on two datasets of laparoscopic cholecystectomy videos, outperforming various state-of-the-art temporal refinement approaches.
BibTeX
@inproceedings{czempiel2021opera,
  author = {Tobias Czempiel and Magdalini Paschali and Daniel Ostler and Seong Tae Kim and Benjamin Busam and Nassir Navab},
  title = {{OperA: Attention-Regularized Transformers for Surgical Phase Recognition}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=x5-iJQsq9WeC},
}
Figure