Deep Placental Vessel Segmentation for Fetoscopic Mosaicking

Sophia Bano, Francisco Vasconcelos, Luke M. Shepherd, Emmanuel B. Vander Poorten, Tom Vercauteren, Sébastien Ourselin, Anna L. David, Jan Deprest, Danail Stoyanov

TL;DR

This paper introduces a novel deep learning framework for segmenting placental blood vessels in fetoscopic videos, a crucial step for treating twin-to-twin transfusion syndrome (TTTS). By leveraging U-Net for accurate vessel segmentation and then using these segmented maps for image registration, the system creates comprehensive placental maps that overcome the limitations of traditional image-based methods, offering improved robustness and reduced drift.

What Problem Does This Paper Address?

Twin-to-twin transfusion syndrome (TTTS) is a serious pregnancy complication where blood flow is unevenly distributed between identical twins. A common treatment involves fetoscopic laser photocoagulation, where a surgeon uses a fetoscope to identify and ablate abnormal vascular connections on the chorionic plate of the placenta. However, this procedure is fraught with challenges: the fetoscopic view is limited, visibility is poor due to amniotic fluid, and the environment is mobile. To aid surgeons, the ideal solution would be an automatically generated, expanded map of the placental vasculature. Existing methods for creating such maps (mosaicking) often struggle with the dynamic and low-visibility conditions of in vivo fetoscopy, leading to drift and inaccuracies. This paper aims to bridge this gap by developing a robust placental vessel segmentation and mosaicking system specifically for fetoscopic videos.

Methods

The proposed framework consists of two main stages:

  1. Placental Vessel Segmentation: The authors employ a U-Net architecture, a popular choice for medical image segmentation due to its effectiveness even with limited training data. They adapt the standard U-Net by using a combined loss function that includes both binary cross-entropy and Intersection over Union (IoU) loss. This hybrid loss function, empirically found to improve accuracy, helps the network learn to accurately identify placental vessels from the challenging fetoscopic images. The output of this stage is not a binary mask but a "vessel probability map," which provides richer information for the subsequent registration step.
  2. Vessel Map Registration and Mosaicking: Instead of registering raw image pixels, the system registers the vessel probability maps generated by the U-Net. This is a key innovation. They use a direct intensity-based registration method, specifically a pyramidal Lucas-Kanade algorithm, applied to these probability maps. This approach is more robust to the poor visibility, lighting variations, and occlusions common in fetoscopic videos, as the vessel maps highlight the relevant structures while downplaying irrelevant visual noise. The registration is performed sequentially, and the transformations are accumulated to create a larger, unified mosaic of the placental vasculature.

Datasets

  • Training and Validation: The authors created a new dataset of 483 manually annotated in vivo fetoscopic images from 6 different TTTS laser ablation videos. These images were carefully selected to exclude occlusions by the fetus or surgical tools. The dataset is made publicly available for research.
  • Evaluation: The framework is evaluated on 6 different in vivo fetoscopic video sequences, some of which contain up to 400 frames.

Key Insights and Ideas

  • Vessel Probability Maps for Registration: The most significant insight is that registering vessel probability maps, rather than raw image intensities, dramatically improves robustness in challenging fetoscopic environments. This is because the maps filter out noise and focus on the critical vascular structures.
  • U-Net's Adaptability: The U-Net architecture, with a combined loss function, proves highly effective for segmenting vessels even with a relatively small, manually annotated dataset.
  • Quantifying Drift Without Ground Truth: The paper introduces a novel method for quantifying registration drift error by analyzing the structural similarity (SSIM) and IoU between overlapping, non-consecutive frames. This is crucial for in vivo scenarios where ground-truth homographies are unavailable.
  • Addressing Occlusions: The vessel-based registration implicitly handles occlusions by focusing on visible vessel segments and effectively "filling in" missing information through the mosaicking process.

Key Results

  • Segmentation Performance: The U-Net architecture, particularly with a ResNet101 backbone, significantly outperforms a previously proposed baseline method (Sadda et al. [22]) in placental vessel segmentation, achieving an average Dice score of 0.78 and IoU of 0.66 across six folds. Qualitative results show the U-Net can detect smaller vessels missed by human annotators.
  • Registration Robustness: Vessel-based registration demonstrates superior performance compared to image-intensity-based registration. It shows better robustness, particularly in longer sequences (up to 400 frames), with significantly reduced drift accumulation.
  • Drift Reduction: The proposed method reduces drift accumulation to a negligible level, making it suitable for creating comprehensive, long-term placental maps.
  • Qualitative Improvement: The generated mosaics provide an expanded field-of-view and can even improve vessel segmentation by blending information from multiple registered frames.

Limitations and Future Work

  • Limited Ground Truth for Registration: The paper acknowledges that quantitative evaluation of registration error in vivo is challenging due to the absence of ground-truth homographies. The proposed drift quantification method is a step towards addressing this, but further validation would be beneficial.
  • Manual Annotation Effort: While the U-Net performs well with a limited dataset, the initial manual annotation of 483 images is a significant undertaking. Future work could explore semi-supervised or unsupervised learning techniques to reduce this burden.
  • Real-time Performance: While the paper focuses on accuracy and robustness, the real-time performance of the entire pipeline for intra-operative use is not explicitly detailed.
  • Generalizability: While tested on 6 different patients, further validation on a larger and more diverse set of fetoscopic videos would strengthen the generalizability of the approach.

The authors suggest that their framework, along with the contributed dataset, provides a strong benchmark for future research in fetoscopic placental vessel segmentation and registration.

Summary updated

Abstract

During fetoscopic laser photocoagulation, a treatment for twin-to-twin transfusion syndrome (TTTS), the clinician first identifies abnormal placental vascular connections and laser ablates them to regulate blood flow in both fetuses. The procedure is challenging due to the mobility of the environment, poor visibility in amniotic fluid, occasional bleeding, and limitations in the fetoscopic field-of-view and image quality. Ideally, anastomotic placental vessels would be automatically identified, segmented and registered to create expanded vessel maps to guide laser ablation, however, such methods have yet to be clinically adopted. We propose a solution utilising the U-Net architecture for performing placental vessel segmentation in fetoscopic videos. The obtained vessel probability maps provide sufficient cues for mosaicking alignment by registering consecutive vessel maps using the direct intensity-based technique. Experiments on 6 different in vivo fetoscopic videos demonstrate that the vessel intensity-based registration outperformed image intensity-based registration approaches showing better robustness in qualitative and quantitative comparison. We additionally reduce drift accumulation to negligible even for sequences with up to 400 frames and we incorporate a scheme for quantifying drift error in the absence of the ground-truth. Our paper provides a benchmark for fetoscopy placental vessel segmentation and registration by contributing the first in vivo vessel segmentation and fetoscopic videos dataset.
BibTeX
@inproceedings{bano2020deep,
  author = {Sophia Bano and Francisco Vasconcelos and Luke M. Shepherd and Emmanuel B. Vander Poorten and Tom Vercauteren and Sébastien Ourselin and Anna L. David and Jan Deprest and Danail Stoyanov},
  title = {{Deep Placental Vessel Segmentation for Fetoscopic Mosaicking}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2020},
  url = {https://openreview.net/forum?id=DC-YXNuSU6P},
}
Figure