TL;DR
This paper introduces a novel two-stage neural framework that leverages surface normal information to significantly improve 3D reconstruction of colonoscopy videos, particularly in challenging "en face" views. By incorporating normal consistency losses and a normal-depth refinement module, the method achieves more accurate and robust depth estimation, leading to better overall 3D surface reconstruction with reduced artifacts.
What Problem Does This Paper Address?
Reconstructing a detailed 3D model of the colon from endoscopic videos is crucial for improving polyp detection and ensuring complete examination. However, this task is notoriously difficult due to varying illumination, reflective surfaces, and challenging viewing angles, especially "en face" views where the camera is perpendicular to the colon surface. These conditions often lead to inaccurate depth predictions and incomplete or noisy 3D reconstructions. Existing methods struggle with these "en face" views, where photometric cues are less informative, and often require extensive post-processing to yield usable results.
Methods
The proposed framework is a two-stage approach:
- Normal-based Depth Initialization: This stage uses a self-supervised neural network trained with a "normal consistency loss." This loss encourages consistency between predicted surface normals across different frames of the same video. This helps to regularize depth predictions, especially in areas with limited geometric texture. This module is inspired by existing self-supervised depth estimation networks but enhanced with the normal consistency objective.
- Normal-Depth Refinement: This stage takes the output from the initialization module and further refines the depth and normal predictions. It leverages the strong correlation between scene illumination and surface geometry in endoscopic environments. This module uses the relationship between illumination and surface normals to recursively refine predictions, aiming to improve accuracy, especially for "en face" views.
The overall framework integrates these two stages, with the initialization providing a strong starting point for the refinement module. The authors also mention using a SLAM-based pipeline for fusing frame-wise outputs into a 3D mesh, following prior work.
Datasets
- Clinical Data: The authors collected videos from 85 clinical colonoscopy procedures. They sampled 185,000 frames for training and 5,000 for validation for the normal-based depth initialization network.
- Colonoscopy 3D Video Dataset (C3DV): This dataset, which provides ground truth depth maps and camera poses for a phantom colon, was used for training the normal-depth refinement module and for evaluating frame-wise depth estimation. The dataset was divided into 5 cross-validation partitions.
Key Insights and Ideas
- Surface Normals as a Powerful Regularizer: The paper strongly demonstrates that surface normal information is not just a byproduct but a critical component for improving depth estimation and 3D reconstruction in challenging endoscopic scenarios.
- Normal Consistency for "En Face" Views: The "normal consistency loss" is particularly effective in improving predictions for "en face" views, where traditional photometric cues are less reliable. This highlights the importance of geometric constraints.
- Two-Stage Refinement is Key: The combination of an initial normal-aware depth estimation followed by a normal-depth refinement stage yields superior results compared to either stage alone or photometric-only methods.
- Illumination-Geometry Correlation: The refinement stage effectively exploits the relationship between illumination and surface normals, a characteristic of endoscopic environments, to enhance depth accuracy.
Key Results
- Frame-wise Depth Accuracy: The proposed method significantly outperforms the baseline Monodepth2 on the C3DV dataset across various error metrics (Abs Rel, Sq Rel, RMSE, log RMSE). For instance, the "ND init + 1xNR" model achieved a RMSE of 7.376, compared to 11.779 for Monodepth2.
- 3D Reconstruction Quality: Qualitative comparisons on both C3DV and clinical data show that the proposed method produces more accurate and less artifact-prone 3D reconstructions. The reconstructions are more cylindrical and cover more of the surveyed surface, especially in challenging sequences.
- Reduced Artifacts: The normal-aware training helps to reduce artifacts, such as the "skirt-shape outlier" observed in sequence (a) of the clinical reconstructions when trained without the normal consistency loss.
- Improved "En Face" Reconstruction: The refinement stage demonstrably reduces high surface curvature in "en face" views, leading to more realistic reconstructions.
Limitations and Future Work
- View Type Separation: The current framework treats "down-the-barrel" and "en face" views separately. The authors acknowledge that colonoscopy videos transition between these view types, and a framework that can seamlessly handle these transitions would be clinically valuable.
- Future Work: The authors suggest investigating a unified framework that can handle transitions between different view types as a significant area for future research.
Summary updated