Deep Homography Prediction for Endoscopic Camera Motion Imitation Learning

Martin Huber, Sébastien Ourselin, Christos Bergeles, Tom VercauterenMICCAI 2023 · OpenReview

TL;DR

This paper introduces a novel imitation learning approach to automate endoscopic camera motion by predicting future camera movements from past video sequences. By learning object-motion-invariant homographies, the method bypasses the need for depth information or explicit tool tracking, demonstrating significant improvements over existing baselines and paving the way for more autonomous robotic surgery.

What Problem Does This Paper Address?

Robot-assisted minimally invasive surgery (RMIS) holds great promise for reducing human error and improving surgical outcomes. However, the complexity of these systems often leads to a steep learning curve for surgeons. Automating camera motion is a crucial step towards increasing autonomy in RMIS, as it can alleviate surgeon fatigue and cognitive load. Previous attempts have relied on rule-based methods (e.g., keeping surgical tools centered) or complex reinforcement learning (RL) approaches that suffer from sample inefficiency and a domain gap between simulation and real-world application. While imitation learning (IL) offers a path to learn from real data, it has been hindered by the lack of readily available image-action pairs for camera motion. This paper addresses this gap by proposing a method to learn camera motion directly from retrospective surgical videos, without requiring explicit annotations or tool tracking.

Methods

The core of the proposed method is an imitation learning framework that predicts future camera motion by learning to estimate projective homographies. Instead of directly predicting the 3x3 homography matrix, the authors represent camera motion as a sequence of four-point displacements in homogeneous coordinates, which has been shown to be more amenable to deep learning.

The pipeline involves:

  • Camera Motion Estimator: A pre-trained estimator (based on a ResNet backbone) is used to generate "pseudo-ground-truth" camera motion (homographies) from retrospective video sequences. This estimator is designed to be efficient and robust to object motion.
  • Importance Sampling: To focus learning on informative sequences, the authors employ an importance sampling strategy. This strategy prioritizes sequences where significant camera motion occurs, effectively filtering out static or less dynamic scenes that are less relevant for learning motion prediction.
  • Data Augmentation: Geometric transforms are applied to the image sequences to encourage the model to learn viewpoint-invariant behaviors. Photometric transforms are avoided to ensure the camera motion estimator operates under optimal conditions.
  • Camera Motion Predictor: A deep neural network (e.g., ResNet-18/34/50) takes a sequence of past frames (recall horizon) as input and predicts the future camera motion (homographies) over a preview horizon. This prediction is trained to match the pseudo-ground-truth generated by the estimator.

A key novelty is the deviation from methods that track surgical tools. Instead, this approach learns to predict camera motion directly from the visual flow, making it more general and less dependent on specific surgical instruments.

Datasets

The authors utilize three publicly available datasets of laparoscopic interventions:

  • Cholec80: A dataset of cholecystectomy (gallbladder removal) videos.
  • HeiChole: Another dataset of cholecystectomy videos.
  • AutoLaparo: A dataset of hysterectomy (uterus removal) videos, notable for including public multi-class motion classification labels.

The datasets are preprocessed by cropping and resizing images to a consistent size (240x320) and then split into training, validation, and testing sets.

Key Insights and Ideas

  • Learning from Retrospective Data: The paper demonstrates that valuable camera motion policies can be learned directly from existing surgical videos, bypassing the need for specialized robotic setups or extensive manual annotation.
  • Object-Motion-Invariant Homographies: By focusing on homographies that are invariant to object motion, the method can effectively disentangle camera movement from the dynamic elements within the surgical scene.
  • Importance Sampling for Efficiency: The observation that a large fraction of surgical videos are static is leveraged through importance sampling. This significantly improves learning efficiency by focusing on the more informative, dynamic segments.
  • No Tool Tracking Required: A significant departure from prior work is the elimination of the need to track surgical tools. This makes the method more generalizable and less susceptible to failures in tool detection.
  • Implicit Domain Adaptation: By learning from real surgical videos, the method inherently addresses the domain gap often encountered in RL approaches that rely on simulations.

Key Results

  • Improved Prediction Accuracy: The proposed method achieves statistically significant improvements over baseline methods (O(1) and O(2) Taylor expansions) on the Cholec80 and HeiChole datasets, with improvements of up to 47% over camera motion continuation.
  • Alignment with Ground Truth: On the AutoLaparo dataset, the predicted camera motion aligns well with the provided multi-class motion labels, demonstrating the method's ability to capture different types of camera movements (left, right, up, down).
  • Efficient Online Estimation: The camera motion estimator, a key component, is shown to be efficient, with a ResNet-34 backbone executing 11x faster than other evaluated methods while maintaining a low GPU memory footprint.

Limitations and Future Work

  • Limited Preview Horizon: The current implementation is limited to a preview horizon of M=1. The authors suggest extending this for model predictive control.
  • Camera Motion Estimator Limitations: The utilized camera motion estimator is efficient and good at isolating object motion from camera motion, but it is limited to relatively small camera motions. Improving this estimator for larger motions would enable longer preview horizons.
  • Auxiliary Tasks: To improve explainability and prediction accuracy, the authors suggest incorporating auxiliary tasks like tool and organ segmentation, and surgical phase recognition.
  • Real-World Deployment: A key future direction is to test the model in a real robotic setup to investigate transferability, potentially fine-tuning the policy with reinforcement learning and human feedback.

Summary updated

Abstract

In this work, we investigate laparoscopic camera motion automation through imitation learning from retrospective videos of laparoscopic interventions. A novel method is introduced that learns to augment a surgeon’s behavior in image space through object motion invariant image registration via homographies. Contrary to existing approaches, no geometric assumptions are made and no depth information is necessary, enabling immediate translation to a robotic setup. Deviating from the dominant approach in the literature which consist of following a surgical tool, we do not handcraft the objective and no priors are imposed on the surgical scene, allowing the method to discover unbiased policies. In this new research field, significant improvements are demonstrated over two baselines on the Cholec80 and HeiChole datasets, showcasing an improvement of \(47\%\) over camera motion continuation. The method is further shown to indeed predict camera motion correctly on the public motion classification labels of the AutoLaparo dataset. All code is made accessible on GitHub (https://github.com/RViMLab/homography_imitation_learning).
BibTeX
@inproceedings{huber2023deep,
  author = {Martin Huber and Sébastien Ourselin and Christos Bergeles and Tom Vercauteren},
  title = {{Deep Homography Prediction for Endoscopic Camera Motion Imitation Learning}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  url = {https://openreview.net/forum?id=qszpQoByuE},
}
Figure