Live Laparoscopic Video Retrieval with Compressed Uncertainty

Tong Yu, Pietro Mascagni, Juan Verde, Jacques Marescaux, Didier Mutter, Nicolas Padoy

Abstract

Searching through large volumes of medical data to retrieve relevant information is a challenging yet crucial task for clinical care. However the primitive and most common approach to retrieval, involving text in the form of keywords, is severely limited when dealing with complex media formats. Content-based retrieval offers a way to overcome this limitation, by using rich media as the query itself. Surgical video-to-video retrieval in particular is a new and largely unexplored research problem with high clinical value, especially in the real-time case: using real-time video hashing, search can be achieved directly inside of the operating room. Indeed, the process of hashing converts large data entries into compact binary arrays or hashes, enabling large-scale search operations at a very fast rate. However, due to fluctuations over the course of a video, not all bits in a given hash are equally reliable. In this work, we propose a method capable of mitigating this uncertainty while maintaining a light computational footprint. We present superior retrieval results (3-4 % top 10 mean average precision) on a multi-task evaluation protocol for surgery, using cholecystectomy phases, bypass phases, and coming from an entirely new dataset introduced here, critical events across six different surgery types. Success on this multi-task benchmark shows the generalizability of our approach for surgical video retrieval.

Introduction

During complex laparoscopic procedures, surgeons often encounter rare anatomical variations or sudden complications. The ability to instantly retrieve similar historical cases from a massive database could provide invaluable intraoperative guidance. However, content-based video retrieval in a live surgical environment presents significant technical hurdles.

Current methods rely on video hashing, which compresses video clips into short binary strings (hashes). While efficient, these hashes struggle with the dynamic nature of live video. As a feed evolves, specific bits in the hash may "flicker" or fluctuate, introducing noise. If a search engine treats every bit as equally certain, these unreliable bits lead to semantically irrelevant results. The core challenge lies in identifying and discounting these untrustworthy bits in real-time without compromising system speed.

Methodology: ULA-CODE

To address these challenges, researchers developed ULA-CODE (Uncertain Look-Ahead Code). This system builds upon a dual-encoder architecture: a Primary Encoder that generates "gold standard" hashes from complete video clips for the database, and a Secondary Encoder that predicts these hashes from partial, live video streams.

The ULA-CODE pipeline, featuring self-supervised training and uncertainty-aware retrieval.

ULA-CODE introduces a formal framework for handling bit uncertainty through two metrics:

  1. Type I Uncertainty: Captures inherent bit instability by measuring fluctuations over the duration of a clip.
  2. Type II Uncertainty: Measures the prediction error (disagreement) between the Primary and Secondary encoders.

To maintain a minimal computational footprint, ULA-CODE does not store full uncertainty scores. Instead, it identifies the $k$ most uncertain bits and stores their positions using a lexicographic ranking compression technique. During retrieval, the system simply ignores these masked bits when calculating similarity, effectively increasing the signal-to-noise ratio.

Data Benchmarks

The model was evaluated on a comprehensive "triple benchmark" representing various levels of surgical complexity:

  • Cholec80: 80 gallbladder removals annotated by surgical phase.
  • Bypass40: 40 gastric bypass videos featuring complex workflows.
  • CEV64 (New): A novel dataset introduced in this work, containing 64 videos across six surgery types. Unlike phase-based datasets, CEV64 focuses on 10 specific intraoperative events.
The CEV64 dataset includes 10 critical intraoperative event types across diverse surgical procedures.

Results and Discussion

ULA-CODE demonstrated a significant performance leap, outperforming the standard LA-CODE baseline by 3–4% in mean Average Precision (mAP@10) across all datasets. The method proved robust, with average performance across hyperparameter ranges consistently exceeding the baseline.

Efficiency is a hallmark of the system. The entire pipeline—from I3D feature extraction to uncertainty-aware search—executes in under one second on a consumer-grade GPU (NVIDIA 1080 Ti), making it truly "real-time ready" for the operating room. Qualitatively, the model successfully distinguished between visually similar but semantically distinct events, such as "active bleeding" versus "tissue division," which often confuse traditional color-based models.

Conclusion and Future Directions

While ULA-CODE represents a major step forward in unsupervised surgical video retrieval, several avenues for improvement remain. Future work could incorporate human-labeled supervision (e.g., instrument or anatomy tags) to further refine retrieval accuracy. Additionally, replacing the general-purpose I3D backbone with a feature extractor pretrained on a massive surgical corpus could yield more domain-specific representations. Finally, integrating broader workflow context—understanding where a clip sits within a multi-hour procedure—could provide even more relevant clinical insights.

Summary updated

Abstract

Searching through large volumes of medical data to retrieve relevant information is a challenging yet crucial task for clinical care. However the primitive and most common approach to retrieval, involving text in the form of keywords, is severely limited when dealing with complex media formats. Content-based retrieval offers a way to overcome this limitation, by using rich media as the query itself. Surgical video-to-video retrieval in particular is a new and largely unexplored research problem with high clinical value, especially in the real-time case: using real-time video hashing, search can be achieved directly inside of the operating room. Indeed, the process of hashing converts large data entries into compact binary arrays or hashes, enabling large-scale search operations at a very fast rate. However, due to fluctuations over the course of a video, not all bits in a given hash are equally reliable. In this work, we propose a method capable of mitigating this uncertainty while maintaining a light computational footprint. We present superior retrieval results (3-4 % top 10 mean average precision) on a multi-task evaluation protocol for surgery, using cholecystectomy phases, bypass phases, and coming from an entirely new dataset introduced here, critical events across six different surgery types. Success on this multi-task benchmark shows the generalizability of our approach for surgical video retrieval.
BibTeX
@article{yu2023live,
  author = {Tong Yu and Pietro Mascagni and Juan Verde and Jacques Marescaux and Didier Mutter and Nicolas Padoy},
  title = {{Live Laparoscopic Video Retrieval with Compressed Uncertainty}},
  journal = {Medical Image Analysis},
  year = {2023},
  url = {https://openreview.net/forum?id=YcM46l3KAw},
}

Loading related content...

Figure