Deep-learning-based endoscopic single-shot fringe projection profilometry

Ruizhi Zuo, Shuwen Wei, Yaning Wang, Ruichen Huang, Waymond Rodgers, Jinglun Yu, Michael H. Hsieh, Axel Krieger, Jin U. KangJournal of Biomedical Optics 2025 · DOI

TL;DR

This paper introduces a novel deep-learning-based endoscopic system for single-shot fringe projection profilometry (FPP), enabling real-time 3D depth map generation. By combining a dual-channel endoscope with a two-part neural network (MaskNet and DepthNet), the system achieves millimeter-scale accuracy at 20 FPS, paving the way for more reliable image-guided robotic surgery.

What Problem Does This Paper Address?

Traditional fringe projection profilometry (FPP) is a powerful technique for generating accurate 3D depth maps. However, its reliance on multiple image acquisitions and precise synchronization makes it slow and prone to motion artifacts, especially when imaging dynamic biological tissues. This limits its application in real-time scenarios like robotic surgery, where precise depth information is crucial for navigation and control. Existing deep learning approaches often focus on synthetic data or require complex hardware setups. This paper aims to bridge this gap by developing a single-shot FPP system integrated into an endoscope that can generate accurate depth maps in real-time, even in challenging clinical environments.

Methods

The core of this work is a novel deep-learning architecture designed for single-shot FPP. The system utilizes a dual-channel endoscope, where one channel projects fringe patterns and the other captures images. This setup simplifies the hardware compared to traditional FPP systems that require separate projectors and cameras.

The depth reconstruction is handled by a two-path neural network:

  • MaskNet: This network segments the target tissue from the background. This is crucial because clinical environments often have complex backgrounds that can resemble tissue, making accurate segmentation challenging.
  • DepthNet: This network predicts the depth map of the segmented tissue.

The outputs of MaskNet and DepthNet are combined through pixel-wise multiplication to produce the final depth map. This approach allows the network to focus on the relevant tissue and ignore background noise, improving accuracy. The authors also developed a data synthesis program using Blender to generate a large and diverse dataset for training, overcoming the limitations of scarce real-world surgical data. They explored both sinusoidal and binary fringe patterns, finding that binary patterns, despite being simpler, yielded better results in their system.

Datasets

  • Synthetic Dataset: Generated using a custom Blender simulation. This dataset was crucial for initial training and optimization of the network, allowing for the generation of a large number of diverse training examples with known ground truth depth maps.
  • Experimental Dataset: Collected using the developed endoscopic FPP system on intestinal phantom samples. This dataset was used for final validation and to assess the system's performance in a more realistic setting.

Key Insights and Ideas

  • Single-Shot FPP is Feasible and Effective: The paper demonstrates that a single fringe pattern image, processed by a deep learning network, can yield accurate depth maps, eliminating the need for multiple acquisitions and synchronization.
  • Dual-Network Architecture (MaskNet + DepthNet) is Key: Separating the segmentation task (MaskNet) from depth prediction (DepthNet) significantly improves performance. MaskNet helps the DepthNet focus on the relevant foreground, leading to more robust and accurate depth maps, especially in complex backgrounds.
  • Binary Patterns Outperform Sinusoidal in this Setup: Counter-intuitively, binary fringe patterns, which are simpler to generate and less prone to certain artifacts (like those from DMD mirror flipping), provided better depth reconstruction accuracy in their system, especially for dynamic measurements.
  • Data Synthesis is Crucial for Training: The use of Blender for generating synthetic data was essential for training a robust deep learning model, especially given the limited availability of real surgical data.

Key Results

  • Real-time Performance: The system achieves a processing time of approximately 12.75 ms per frame, enabling a frame rate of around 78 FPS, which is faster than the image acquisition rate of 50 ms (20 FPS).
  • High Accuracy: The system achieved a maximum depth prediction error of ~2 mm with an absolute relative error of 2% to 3% on both synthetic and experimental data.
  • Optimal Frequency: For their setup, 20 Hz was found to be the optimal fringe pattern frequency, balancing low-frequency sensitivity to details with high-frequency aliasing issues.
  • MaskNet's Impact: Ablation studies showed that removing MaskNet led to a noticeable decrease in depth map prediction accuracy, highlighting its importance for focusing on the target object.

Limitations and Future Work

  • Limited Field of View and Noise: The compact endoscopic design results in a limited field of view, potentially leading to lower numerical aperture and increased image noise. This can affect depth resolution.
  • Ground Truth Generation: The accuracy of the ground truth depth maps, generated using conventional FPP, can be affected by image quality and non-uniform lighting, which can in turn impact network training.
  • Complex Tissue Structures: While the system performed well on relatively simple intestinal tissue, future work will focus on extending its capabilities to more complex anatomical structures and layered tissues.
  • Improving Optical Setup: Further optimization of the optical setup, potentially by replacing the projector with a simpler LED light source and a fixed binary mask, could further improve system stability and performance.
  • More Accurate Ground Truth: Exploring more accurate ground truth generation methods, such as computed tomography or optical coherence tomography, could lead to even better model performance.

Summary updated

Abstract

A deep-learning-based single-shot FPP endoscopic system was shown to be highly effective in real-time depth map generation with millimeter-scale error. Implementing such a system has the potential to improve the reliability of image-guided robotic surgery.
BibTeX
@article{zuo2025deep,
  author = {Ruizhi Zuo and Shuwen Wei and Yaning Wang and Ruichen Huang and Waymond Rodgers and Jinglun Yu and Michael H. Hsieh and Axel Krieger and Jin U. Kang},
  title = {{Deep-learning-based endoscopic single-shot fringe projection profilometry}},
  journal = {Journal of Biomedical Optics},
  year = {2025},
  doi = {10.1117/1.jbo.30.8.086003},
}
Figure