Motion-based camera localization system in colonoscopy videos

Heming Yao, Ryan W. Stidham, Zijun Gao, Jonathan Gryak, Kayvan NajarianMedical Image Analysis 2021 · DOI

TL;DR

This paper introduces a novel vision-based system for automatically localizing the camera within colonoscopy videos. By estimating camera motion and classifying anatomical segments, it aims to standardize colonoscopy analysis and provide crucial contextual information for diagnosis and treatment monitoring, outperforming existing methods.

What Problem Does This Paper Address?

Optical colonoscopy is a vital diagnostic tool, but interpreting findings can be subjective and vary between observers. To address this, automated analysis is desirable. A key challenge is understanding the camera's position within the colon to provide anatomical context for lesions, bleeding sources, or disease severity. Existing methods often rely on external sensors or are limited in their validation on real-world colonoscopy data. This paper tackles the problem of robustly and accurately estimating camera location and classifying colon segments using only the video feed, making it more accessible and cost-effective.

Methods

The core of the system is a self-supervised deep learning framework for camera motion estimation. Key innovations include:

  • Optical Flow Input: Incorporating optical flow between consecutive frames provides explicit motion information, improving robustness to video noise.
  • Specular Region Masking: A novel approach to estimate and filter out specular regions (shiny areas caused by moisture) that can disrupt photometric error calculations, focusing the network on more reliable surface information.
  • Movement Consistency Loss: A new loss term that enforces consistency between forward and backward camera motion estimations, enhancing the network's generalization.
  • Anatomical Colon Segment Classification: After estimating camera motion and deriving a "location index" (a normalized measure of progress through the colon), a colon template is constructed. This template allows for classification of frames into anatomical segments (cecum, ascending colon, etc.), providing contextual information.

Datasets

  • Image Classification Dataset: 34,810 frames from 29 colonoscopy videos, manually annotated for non-informative frames and frames with biopsy forceps.
  • Localization Dataset: 44 colonoscopy videos, split into:
  • Set 1 (16 videos): For building the camera motion estimation network.
  • Set 2 (18 videos): For colon template building.
  • Set 3 (10 videos): Paired with ScopeGuide data for independent evaluation.
  • EndoSLAM Dataset: An external dataset with ground truth for camera pose, used for validating the motion estimation algorithm.

Key Insights and Ideas

  • Specular regions are a major hurdle: The presence of moisture in the colon creates specular reflections that significantly degrade the performance of traditional photometric loss-based methods. The proposed specular masking technique is crucial for overcoming this.
  • Consistency is key for robustness: Enforcing consistency between forward and backward motion estimations significantly improves the generalization of the camera motion estimation.
  • Location index outperforms time index: Using a derived "location index" (representing progress through the colon) for anatomical segment classification yields substantially better results than simply using the time index of frames. This highlights the importance of accurate spatial understanding.
  • Vision-only is feasible and effective: The system demonstrates that accurate camera localization and anatomical segmentation can be achieved using only video data, without requiring additional sensors, making it highly practical for clinical integration.

Key Results

  • Superior Motion Estimation: The proposed method outperformed other published methods on the EndoSLAM dataset, achieving lower Absolute Trajectory Error (ATE) and Relative Pose Error (RPE Rot.).
  • Accurate Anatomical Classification: The system achieved an average accuracy of 0.754 for anatomical colon segment classification on real-world colonoscopy videos, significantly higher than baseline methods using time index or ScopeGuide length.
  • Robustness to Noise and Variations: The inclusion of optical flow and movement consistency loss improved the robustness of the motion estimation.
  • EndoSLAM Validation: The proposed method showed good generalization across high-resolution and low-resolution videos from the EndoSLAM dataset.

Limitations and Future Work

  • Manual Withdrawal Time Annotation: Currently, the time at which the camera withdrawal begins is manually annotated. Future work aims to develop an image classifier to automatically detect the cecum/ileum to enable fully automated localization.
  • Colon Length Assumption: The anatomical segment classification assumes a generic colon length. Future work plans to integrate patient-specific information, surgical history, and automatically detected anatomical features (like the appendiceal orifice) to improve segment identification.
  • Scale Drifting: While the scale-drifting problem inherent in self-supervised monocular methods appears minor in their results, future work will explore integrating additional information to further mitigate this.
  • Anatomical Feature Integration: The authors suggest leveraging detected anatomical features (cecum, transverse colon, rectum) to refine location index estimation.

Summary updated

BibTeX
@article{yao2021motion,
  author = {Heming Yao and Ryan W. Stidham and Zijun Gao and Jonathan Gryak and Kayvan Najarian},
  title = {{Motion-based camera localization system in colonoscopy videos}},
  journal = {Medical Image Analysis},
  year = {2021},
  doi = {10.1016/j.media.2021.102180},
}
Figure