Live laparoscopic video retrieval with compressed uncertainty

Tong Yu, Pietro Mascagni, Juan Verde, Jacques Marescaux, Didier Mutter, Nicolas Padoy

Abstract

This paper presents an unsupervised hashing method for real-time video retrieval from live laparoscopic sources. By introducing and formalizing the concept of uncertainty in video hashing, the proposed system anticipates future content to find similar past procedures. The method demonstrates strong performance across a multi-task, multi-procedure evaluation setup, including a study of surgical events across six different types of laparoscopic surgery.

Introduction

Laparoscopic surgery is a high-stakes environment where surgeons often face unexpected anatomical variations or rare complications. The ability to instantly retrieve video clips of similar past cases could provide invaluable intraoperative guidance. However, real-time retrieval requires converting high-dimensional video into compact binary "hashes" that can be searched instantly.

In a live setting, these hashes are inherently unstable because the video is still unfolding. This paper addresses the challenge of "uncertain bits"—bits in the hash that fluctuate as more frames arrive—which typically degrade search accuracy. Previous methods treated every bit as 100% reliable; in contrast, this work identifies and discounts untrustworthy bits to improve retrieval quality without sacrificing speed.

Methods

The authors propose ULA-CODE (Uncertain LA-CODE), an extension of the teacher-student Look-Ahead Code architecture. While the "primary" encoder processes complete historical videos to build a database, a "secondary" encoder anticipates future content from truncated live streams during the operation.

ULA-CODE introduces a novel way to handle the resulting uncertainty:

  1. Uncertainty Quantification: The system tracks two metrics: how much a specific bit flips over time (Type I) and how often the live and full-video encoders disagree (Type II).
  2. Uncertainty Compression: To avoid doubling database size with binary masks, the authors use lexicographic unranking. This combinatorial technique stores the specific pattern of the $K$ most uncertain bits as a single compressed rank, drastically reducing memory overhead.
  3. Discounted Search: During retrieval, the system unpacks these masks and mathematically discounts untrustworthy bits when calculating Hamming distances between the live query and the database.
System architecture of ULA-CODE showing training, database encoding, and uncertainty-aware search.

Datasets and Evaluation

The model is trained entirely unsupervised on the Endocorpus dataset, which contains approximately 3,700 hours of unannotated surgical video. Evaluation is conducted on three distinct benchmarks to prove generalizability:

  • Cholec80: 80 cholecystectomy videos annotated with surgical phases.
  • Bypass40: 40 gastric bypass videos.
  • CEV64 (Collected Events in Surgery): A new benchmark introduced in this paper containing 10 critical intraoperative events (e.g., active bleeding, mesh placement) across 6 different surgery types.
Visualization of the multi-task evaluation setup across different surgical procedures and datasets.

Key Results

  • Accuracy Boost: ULA-CODE consistently improves retrieval accuracy, showing a 3-4% boost in Mean Average Precision (mAP@10) over baseline methods across all datasets.
  • Efficiency: Despite the added complexity of uncertainty tracking, the entire inference pipeline (feature extraction, hashing, and search) takes less than 1 second, making it perfectly viable for live use in the operating room.
  • Flexibility: Because the model is unsupervised, it successfully retrieves both broad surgical phases and highly specific, split-second critical events like active bleeding.

Conclusion and Future Work

By recognizing that not all bits in a video hash are created equal, ULA-CODE enables reliable, real-time surgical video retrieval. This "reverse image search" for the operating room provides a foundation for intraoperative decision support. Future work will explore more nuanced clinical evaluation protocols and domain-specific pretraining to further refine the system's visual understanding of surgical scenes.

Summary updated

Abstract

Highlights•Unsupervised hashing method for video retrieval from live sources using anticipation•Introduction of live video retrieval to the domain of surgery.•Introduction and formalization of the concept of uncertainty in video hashing•Strong performance on a multi-task, multi-procedure evaluation setup•Study of surgical events found across 6 different types of laparoscopic surgery.
BibTeX
@article{yu2023live,
  author = {Tong Yu and Pietro Mascagni and Juan Verde and Jacques Marescaux and Didier Mutter and Nicolas Padoy},
  title = {{Live laparoscopic video retrieval with compressed uncertainty}},
  journal = {Medical Image Analysis},
  year = {2023},
  url = {https://openreview.net/forum?id=YcM46l3KAw},
}
Figure