High-fidelity Endoscopic Image Synthesis by Utilizing Depth-guided Neural Surfaces

Baoru Huang, Yida Wang, Anh Nguyen, Daniel S. Elson, Francisco Vasconcelos, Danail Stoyanov

Abstract

In surgical oncology, screening colonoscopy is vital for diagnostic assistance and surgical navigation. However, accurate 3D surface reconstruction from monocular endoscopic videos remains challenging due to constrained camera movement, inadequate tissue texture, and scale ambiguity. We introduce a novel method for colon section reconstruction by leveraging Neural Implicit Surfaces (NeuS) supplemented by a single frame of depth map. By utilizing a cross-domain depth estimator and an adaptive regularization technique, our approach achieves stable, consistently scaled reconstructions. Validation on phantom imagery demonstrates exceptional accuracy in rendering colon sections, even capturing unseen surface portions, promising enhanced quality in cancer screening and treatment interventions.

Introduction

Colonoscopy is the gold standard for colorectal cancer screening, yet navigating the "pink tunnel" of the colon is notoriously difficult. Surgeons often lack a clear 3D understanding of the environment, which can lead to missed polyps or incomplete examinations. While Neural Radiance Fields (NeRF) and Neural Implicit Surfaces (NeuS) have revolutionized 3D reconstruction, they struggle in endoscopy due to the featureless nature of colon tissue, constrained camera paths, and the "confused scale" problem inherent in monocular vision.

Methodology

Our solution builds upon Neural Implicit Surfaces (NeuS), representing 3D surfaces as a Signed Distance Function (SDF). We introduce three key innovations to address the specific challenges of endoscopic imaging:

Cross-Domain Depth Guidance

To avoid the need for massive labeled medical datasets, we utilize a DPT Hybrid model pre-trained on natural scenes. These depth predictions are rescaled to the specific endoscopic scene using a sparse point cloud generated via Structure-from-Motion (COLMAP).

Single-Frame Anchoring

We demonstrate that even a single frame of depth information can act as a powerful prior to stabilize the entire reconstruction and resolve scale issues across the sequence.

Adaptive Eikonal Regularization

Standard SDF training uses a fixed Eikonal loss, which often leads to over-smoothed or "flat" reconstructions in low-texture areas. We introduced a ray-wise weight ($\lambda_r$) that adaptively relaxes this constraint based on the distance between the predicted depth and the current surface estimate. This allows the model to capture finer geometric details while maintaining mathematical validity.

Qualitative comparison showing the MVS observation, our reconstruction, and the ground truth.

Experimental Results

We evaluated our model using the C3VD (Colonoscopy 3D Video Dataset), which contains high-resolution video sequences from four distinct colon phantom regions: cecum, descending, sigmoid, and transcending.

Quantitative Performance

Our method was benchmarked against Gaussian NeRF and Vanilla NeRF. As shown in the table below, our approach consistently outperformed or matched the baselines in terms of Peak Signal-to-Noise Ratio (PSNR) across 11 different sub-sequences. Notably, in the "c1_a" sequence, we achieved a PSNR of 34.27, significantly higher than the 32.84 and 32.71 achieved by Gaussian and Vanilla NeRF, respectively.

Quantitative PSNR comparison across various colon sequences against baseline methods.

Qualitative Performance

Visual inspections reveal that our method captures the complex folds of the colon wall with high fidelity. Because the model learns a continuous implicit field, it can effectively "fill in the gaps" and render portions of the colon surface that were not directly captured in the original video frames, a feat difficult for traditional MVS approaches.

Discussion and Conclusion

This work demonstrates that depth estimators trained on everyday objects can provide a useful geometric skeleton for medical tasks when properly rescaled. By making regularization adaptive, we overcome the geometric bias that typically plagues neural surfaces in low-texture environments.

Future work will focus on addressing the static scene assumption, as the real colon is a deformable organ. Additionally, we aim to reduce the dependency on COLMAP for initial scale alignment and validate the model on clinical data containing fluids and surgical tools. Despite these challenges, this breakthrough opens new avenues for achieving stable and consistently scaled reconstructions in clinical practice.

Summary updated

Abstract

In surgical oncology, screening colonoscopy plays a pivotal role in providing diagnostic assistance, such as biopsy, and facilitating surgical navigation, particularly in polyp detection. Computer-assisted endoscopic surgery has recently gained attention and amalgamated various 3D computer vision techniques, including camera localization, depth estimation, surface reconstruction, etc. Neural Radiance Fields (NeRFs) and Neural Implicit Surfaces (NeuS) have emerged as promising methodologies for deriving accurate 3D surface models from sets of registered images, addressing the limitations of existing colon reconstruction approaches stemming from constrained camera movement. However, the inadequate tissue texture representation and confused scale problem in monocular colonoscopic image reconstruction still impede the progress of the final rendering results. In this paper, we introduce a novel method for colon section reconstruction by leveraging NeuS applied to endoscopic images, supplemented by a single frame of depth map. Notably, we pioneered the exploration of utilizing only one frame depth map in photorealistic reconstruction and neural rendering applications while this single depth map can be easily obtainable from other monocular depth estimation networks with an object scale. Through rigorous experimentation and validation on phantom imagery, our approach demonstrates exceptional accuracy in completely rendering colon sections, even capturing unseen portions of the surface. This breakthrough opens avenues for achieving stable and consistently scaled reconstructions, promising enhanced quality in cancer screening procedures and treatment interventions.
BibTeX
@misc{huang2024high,
  author = {Baoru Huang and Yida Wang and Anh Nguyen and Daniel S. Elson and Francisco Vasconcelos and Danail Stoyanov},
  title = {{High-fidelity Endoscopic Image Synthesis by Utilizing Depth-guided Neural Surfaces}},
  year = {2024},
  eprint = {2404.13437},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.13437},
}
Figure