PEg TRAnsfer Workflow Recognition Challenge Report: Do Multi-Modal Data Improve Recognition?

Arnaud Huaulmé, Kanako Harada, Quang-Minh Nguyen, Bogyu Park, Seungbum Hong, Min-Kook Choi, Michael Peven, Yunshuang Li, Yonghao Long, Qi Dou, Satyadwyoom Kumar, Seenivasan Lalithkumar, Hongliang Ren, Hiroki Matsuzaki, Yuto Ishikawa, Yuriko Harai, Satoshi Kondo, Mamoru Mitsuishi, Pierre JanninSSRN 2022 · DOI

TL;DR

This paper reports on the PETRAW challenge, which investigated how different data modalities (video, kinematics, semantic segmentation) impact surgical workflow recognition accuracy. The key takeaway is that combining video and kinematic data significantly improves recognition performance, though at the cost of increased computation time.

What Problem Does This Paper Address?

Accurate, real-time surgical workflow recognition is crucial for developing context-aware computer-assisted surgical systems. While surgical video has been the primary data source, the increasing availability of kinematic data from robot-assisted surgery presents an opportunity to enhance recognition. However, the added value of these new modalities, especially when combined, has not been thoroughly studied. This paper addresses this gap by organizing a challenge to develop and evaluate multi-modal surgical workflow recognition methods, specifically focusing on the peg transfer task.

Methods

The PETRAW challenge involved five tasks designed to assess unimodal and multimodal recognition approaches. Participants were tasked with developing models to recognize surgical workflow at three granularities: phase, step, and activity. The challenge encouraged the use of deep learning techniques, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs) like LSTMs, and Transformers. For multimodal tasks, teams employed various fusion strategies to combine features from different modalities. A key aspect was the use of "application-dependent" accuracy (AD-Accuracy) as the evaluation metric, which accounts for acceptable delays in recognition to better reflect clinical relevance.

Datasets

The challenge utilized a newly created dataset of 150 peg transfer sequences performed on a virtual simulator. This dataset included:

  • Video data: High-resolution video recordings of the surgical procedures.
  • Kinematic data: Synchronously acquired data including instrument position, rotation, forceps aperture, linear and angular velocities.
  • Semantic segmentation data: Pixel-level annotations for different surgical components.
  • Workflow annotations: Ground truth labels for phases, steps, and activities.

This dataset is publicly available to encourage further research.

Key Insights and Ideas

  • Multimodal fusion is king: The most significant finding is that combining multiple data modalities, particularly video and kinematic data, consistently leads to higher recognition accuracy compared to using any single modality alone.
  • Kinematics offer a sweet spot: While video provides rich visual information, kinematic data offers a compelling balance between accuracy gains and computational efficiency, making it a strong candidate for real-time applications.
  • Segmentation's role is nuanced: Semantic segmentation data, while valuable, did not always improve performance when added to video and kinematic data, and in some cases, even decreased it. This suggests that the quality and integration of segmentation are critical.
  • Causality matters, but at a cost: Developing causal models (that don't use future data) is important for real-time applications. While causal methods generally showed slightly lower performance than their non-causal counterparts, their results were still competitive, indicating promising progress.
  • Dataset variability is a challenge: Even with a controlled task like peg transfer, specific sequences exhibited significantly lower performance across all models, highlighting the inherent complexities and potential limitations of even well-annotated datasets.

Key Results

  • Top performance: The best results were achieved by combining video and kinematic data, with AD-Accuracy scores ranging from 93% to 90% for the top-performing teams.
  • Unimodal performance: Unimodal methods achieved respectable results, with video-based recognition generally outperforming kinematic-based recognition in isolation.
  • Computational cost: Multimodal methods, especially those involving video and kinematics, required significantly longer execution times (2,000% to 20,000% increase) compared to unimodal kinematic-based methods, raising questions about the trade-off between accuracy and efficiency.
  • Segmentation impact: The addition of semantic segmentation data to video and kinematic data did not consistently improve performance and, in some cases, led to a decrease.

Limitations and Future Work

  • Dataset limitations: The paper acknowledges that the peg transfer task, while controlled, is simpler than real surgical interventions, with a static environment and a single operator, leading to lower data variability. Future work should include sequences performed by multiple operators and across different systems.
  • Computational cost vs. accuracy: The significant increase in execution time for multimodal methods raises questions about the practical trade-offs for real-time surgical applications. Future work should investigate this further, considering training time and data storage as well.
  • Segmentation quality: The impact of segmentation quality on workflow recognition needs further investigation. Future research could explore different combinations of segmentation and workflow recognition models to better understand this relationship.
  • Causal model development: While promising, the development of truly causal and high-performing models for real-time surgical applications remains an active area of research.

Summary updated

BibTeX
@misc{huaulme2022peg,
  author = {Arnaud Huaulmé and Kanako Harada and Quang-Minh Nguyen and Bogyu Park and Seungbum Hong and Min-Kook Choi and Michael Peven and Yunshuang Li and Yonghao Long and Qi Dou and Satyadwyoom Kumar and Seenivasan Lalithkumar and Hongliang Ren and Hiroki Matsuzaki and Yuto Ishikawa and Yuriko Harai and Satoshi Kondo and Mamoru Mitsuishi and Pierre Jannin},
  title = {{PEg TRAnsfer Workflow Recognition Challenge Report: Do Multi-Modal Data Improve Recognition?}},
  year = {2022},
  doi = {10.2139/ssrn.4088403},
  howpublished = {SSRN},
}
Figure