Rethinking Anticipation Tasks: Uncertainty-Aware Anticipation of Sparse Surgical Instrument Usage for Context-Aware Assistance

Dominik Rivoir, Sebastian Bodenstedt, Isabel Funke, Felix von Bechtolsheim, Marius Distler, Jürgen Weitz, Stefanie Speidel

TL;DR

This paper introduces a novel framework for anticipating the usage of surgical instruments in long laparoscopic videos. By framing anticipation as a regression of "time-to-occurrence" and employing Bayesian Deep Learning, the authors successfully predict sparse instrument events while providing a measure of confidence (uncertainty) that helps identify surgical "trigger events."

What Problem Does This Paper Address?

In the operating room, context-aware AI could revolutionize surgery by preparing instruments ahead of time or assisting robotic systems. However, predicting the future in surgery is difficult because:

  1. Sparsity: Specific instruments (like a specimen bag or scissors) might only appear for a few minutes in an hour-long procedure.
  2. Timing: Most "action anticipation" research focuses on short horizons (e.g., what happens in the next 1 second). In surgery, we need to look minutes ahead.
  3. Data Requirements: Existing long-horizon models often require dense, frame-by-frame activity labels, which are incredibly tedious and expensive to produce.

The authors address these gaps by proposing a model that learns from sparse annotations and handles the inherent "maybe" of the future through uncertainty quantification.

Methods

The authors rethink the anticipation task as a regression problem: given the current video frame, how many minutes ($t$) until instrument $X$ appears?

Key technical components include:

  • Truncated Regression: They limit the prediction horizon to $h$ minutes (e.g., 3, 5, or 7 minutes). If an instrument isn't expected soon, the model predicts a constant value ($h$), allowing it to remain "idle" during irrelevant parts of the surgery.
  • Bayesian Neural Network: They use a Bayesian AlexNet + LSTM architecture. By using Monte-Carlo (MC) Dropout during inference, the model doesn't just give a single prediction; it produces a distribution. This allows the calculation of Epistemic uncertainty (model's lack of knowledge) and Aleatoric uncertainty (inherent randomness in the data).
  • Multi-task Learning: To regularize the model, they add a classification head that categorizes the current state as anticipating, present, or background.

Datasets

The study utilizes the Cholec80 dataset, which contains 80 videos of gallbladder removals (cholecystectomies).

  • Focus: They target 5 sparsely used instruments: bipolar forceps, scissors, clipper, irrigator, and specimen bag.
  • Exclusions: Common tools like the grasper and hook are excluded because they are used almost constantly, making "anticipation" trivial and less useful.

Key Insights and Ideas

  • Trigger Events as Uncertainty Reducers: One of the most compelling ideas is using uncertainty to find "trigger events." For example, in a gallbladder removal, using a clipper is a strong indicator that scissors will be used next. The authors show that when the clipper appears, the model’s uncertainty regarding the future use of scissors drops sharply.
  • Sparse is Enough: The model performs competitively even without "rich" surgical phase annotations. This suggests that visual cues and temporal patterns in instrument usage are sufficient for anticipation, significantly lowering the barrier for training data.
  • Filtering for Precision: By using the model’s own uncertainty scores, the authors can filter out "low-confidence" guesses. This significantly improves the precision of the predictions (measured via pMAE), which is vital for building trust in a clinical setting.

Key Results

  • Performance: The proposed model consistently outperformed "MeanHist" (a baseline based on average instrument timing) across various horizons (2, 3, 5, and 7 minutes).
  • Accuracy: For a 3-minute horizon, the model achieved a weighted Mean Absolute Error (wMAE) of 0.64 minutes.
  • Comparison: The model was competitive with a variant that had access to "Phase" information (knowing exactly which stage of the surgery was occurring), proving the robustness of the visual-only approach.
  • Predictability: Scissors and specimen bags were the most predictable instruments, likely because they are tied to very specific, sequential steps in the surgical workflow.

Limitations and Future Work

  • Dynamic Tools: Instruments like the irrigator and bipolar forceps remain difficult to predict. These are often used "on-demand" (e.g., to clean up a sudden bleeding), meaning they aren't always tied to a predictable workflow step.
  • Aleatoric Uncertainty in Regression: The authors noted that while aleatoric uncertainty was useful for classification, it was less effective for the regression task, suggesting a need for more sophisticated probabilistic heads (like Gaussian Mixture Models).
  • Future Directions: The authors suggest investigating more effective methods for uncertainty quantification and exploring how these predictions could be integrated into actual robotic control loops or OR orchestration systems.

Summary updated

Abstract

Intra-operative anticipation of instrument usage is a necessary component for context-aware assistance in surgery, e.g. for instrument preparation or semi-automation of robotic tasks. However, the sparsity of instrument occurrences in long videos poses a challenge. Current approaches are limited as they assume knowledge on the timing of future actions or require dense temporal segmentations during training and inference. We propose a novel learning task for anticipation of instrument usage in laparoscopic videos that overcomes these limitations. During training, only sparse instrument annotations are required and inference is done solely on image data. We train a probabilistic model to address the uncertainty associated with future events. Our approach outperforms several baselines and is competitive to a variant using richer annotations. We demonstrate the model's ability to quantify task-relevant uncertainties. To the best of our knowledge, we are the first to propose a method for anticipating instruments in surgery.
BibTeX
@inproceedings{rivoir2020rethinking,
  author = {Dominik Rivoir and Sebastian Bodenstedt and Isabel Funke and Felix von Bechtolsheim and Marius Distler and Jürgen Weitz and Stefanie Speidel},
  title = {{Rethinking Anticipation Tasks: Uncertainty-Aware Anticipation of Sparse Surgical Instrument Usage for Context-Aware Assistance}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2020},
  url = {https://openreview.net/forum?id=aZqSoe1gwVXq},
}
Figure