Robust fetoscopic mosaicking from deep learned flow fields

Oluwatosin Alabi, Sophia Bano, Francisco Vasconcelos, Anna L. David, Jan Deprest, Danail StoyanovInternational Journal of Computer Assisted Radiology and Surgery 2022 · DOI

TL;DR

This paper introduces a novel fetoscopic mosaicking pipeline that leverages deep learning-based optical flow (FlowNet-2) and robust estimation (RANSAC) to create seamless, wide-view mosaics of the placenta during fetal surgery. This approach overcomes limitations of existing methods that rely on visible blood vessels, enabling more reliable navigation and identification of abnormalities in challenging surgical conditions.

What Problem Does This Paper Address?

Twin-to-twin transfusion syndrome (TTTS) is a serious pregnancy complication requiring fetoscopic laser photocoagulation (FLP) to treat abnormal placental vascular connections. During FLP, surgeons use a fetoscope, which offers a very limited field of view. This limitation can lead to missed anastomoses and incomplete treatment. To address this, researchers aim to create a wider view of the placenta by stitching together multiple video frames into a mosaic. However, existing mosaicking techniques struggle with the unique challenges of the fetoscopic environment: poor illumination, low contrast, floating particles, and specular reflections, especially when blood vessels, which are often used for registration, are not clearly visible or are too thin to reliably detect. This paper tackles the problem of creating robust and consistent fetoscopic mosaics, particularly in these difficult scenarios where current methods fail.

Methods

The proposed method builds a fetoscopic mosaic by estimating the optical flow between consecutive video frames. Here's a breakdown of their approach:

  • Deep Learning Optical Flow (FlowNet-2): Instead of relying on traditional feature matching or vessel segmentation, the core of their method uses FlowNet-2, a deep learning architecture pre-trained on a large synthetic dataset ("flying chairs"). While this dataset is not directly representative of fetoscopic images, the network learns fundamental geometric relationships that generalize well. This provides dense, pixel-to-pixel correspondences between frames.
  • Robust Estimation (RANSAC): Optical flow can be noisy due to floating particles and specular reflections, which are inconsistent with global camera motion. To filter these outliers, they employ the Random Sample Consensus (RANSAC) algorithm. RANSAC identifies a consistent global affine transformation by iteratively fitting a model to a subset of the data and rejecting points that deviate significantly.
  • Iterative Refinement (Levenberg-Marquardt): After RANSAC identifies inlier correspondences, the Levenberg-Marquardt (LM) algorithm is used to minimize the re-projection error of these inliers, leading to a more precise estimation of the pairwise affine transformation between frames.
  • Mosaic Construction: The final mosaic is built incrementally by applying these estimated pairwise affine transformations.

This pipeline is novel in its combination of a powerful deep learning optical flow backbone with robust outlier filtering, specifically designed to handle the challenges of fetoscopic imaging without relying on explicit vessel detection.

Datasets

The authors used an extended version of the fetoscopy placenta dataset previously introduced in [3]. This extended dataset consists of 6 in vivo TTTS video sequences, with each sequence containing a varying number of frames (ranging from 150 to 420 frames). The dataset is characterized by diverse conditions, including varying levels of occlusion, texture, lighting, and the presence of floating particles. The extended dataset has been made publicly available for reproducibility.

Key Insights and Ideas

  • Optical Flow as a General Registration Tool: The paper demonstrates that deep learning-based optical flow, even when pre-trained on non-medical data, can be a powerful and generalizable tool for fetoscopic image registration, overcoming the limitations of methods that strictly rely on visual features like blood vessels.
  • Robustness Through Outlier Filtering: The combination of optical flow with RANSAC is crucial. It effectively filters out spurious motion cues caused by artifacts like floating particles and specular reflections, which are common in fetoscopic videos and can easily lead to registration failures in simpler optical flow methods.
  • Vessel Visibility is Not Always Necessary: A key takeaway is that their method can produce consistent mosaics even when blood vessels are poorly visible or too thin to be reliably segmented. This significantly expands the applicability of fetoscopic mosaicking to a wider range of surgical scenarios.
  • Pretrained Models Suffice: The authors found that using pre-trained FlowNet-2 models was sufficient for their task, even without fine-tuning on fetoscopic data. This is important given the lack of ground truth motion data for in vivo fetoscopic videos, which would be required for supervised fine-tuning.

Key Results

  • Quantitative Performance: The proposed method consistently outperformed the state-of-the-art vessel-based approach and the classic Lucas-Kanade (LK) method in terms of structural similarity index metric (SSIM) for drift analysis, particularly in videos with challenging visual conditions. For instance, in Video 4, the vessel-based method failed after frame 106, while the proposed method maintained tracking for the entire sequence.
  • Qualitative Performance: Visualizations show that the proposed method produces more consistent and seamless mosaics, especially in videos where vessel segmentation struggles. The paper highlights cases where the vessel-based method produced severely distorted or inconsistent mosaics due to registration failures, while their method maintained coherence.
  • Robustness to Outliers: The analysis of RANSAC outliers confirmed that they generally correspond to floating particles and specular reflections, demonstrating the method's ability to effectively reject these disruptive elements.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while their method is robust, it can still experience some drift over very long sequences.
  • The current approach is sequential, meaning errors can accumulate over time.
  • Future Work:
    • The authors propose developing a hybrid solution that can dynamically switch between a vessel-based approach and an optical flow approach depending on the most suitable context.
  • They also plan to explore global optimization techniques and error drift reduction methods to achieve even greater long-term consistency in large-scale fetoscopic mosaicking.

Summary updated

Abstract

Through analysis of our results, we show that our method outperforms both the vessel-based state of the art and LK, noticeably when vessels are either poorly visible or too thin to be reliably identified. Our approach is thus able to build consistent placental vessel mosaics in challenging cases where currently available alternatives fail.
BibTeX
@article{alabi2022robust,
  author = {Oluwatosin Alabi and Sophia Bano and Francisco Vasconcelos and Anna L. David and Jan Deprest and Danail Stoyanov},
  title = {{Robust fetoscopic mosaicking from deep learned flow fields}},
  journal = {International Journal of Computer Assisted Radiology and Surgery},
  year = {2022},
  doi = {10.1007/s11548-022-02623-1},
}
Figure