DG-Recon: Depth-Guided Neural 3D Scene Reconstruction

Jihong Ju, Ching Wei Tseng, Oleksandr Bailo, Georgi Dikov, Mohsen GhafoorianICCV 2023 · DOI

Introduction

Reconstructing 3D scenes from monocular video sequences is a fundamental challenge in computer vision, with significant implications for robotics and augmented reality. Traditional neural reconstruction methods typically back-project 2D image features into a 3D volumetric grid. However, these approaches often suffer from two primary limitations: "ghosting" artifacts and computational bottlenecks in multi-view fusion.

Ghosting occurs when features are projected along entire camera rays without depth constraints, populating empty space with noise and leading to over-smoothed results. Meanwhile, high-quality fusion techniques like self-attention scale quadratically ($O(K^2)$) with the number of views, hindering real-time performance on edge devices as more camera data is integrated.

Methodology

DG-Recon addresses these challenges through a sparse, depth-guided architecture designed for both efficiency and accuracy. The system introduces three core innovations:

  1. Depth-Guided Back Projection: The system utilizes a lightweight monocular depth estimation network to provide initial geometric priors. Features are only projected into voxels near the estimated surfaces. This creates a sparse 3D representation early in the pipeline, significantly reducing memory overhead and preventing the accumulation of irrelevant data in occluded or empty regions.
  2. Linear Cross-Attention Fusion: To overcome the fusion bottleneck, the system introduces a recursive cross-attention mechanism. By using a persistent 3D feature volume as a query to integrate new 2D features, the complexity is reduced to $O(K)$ relative to the number of views. This allows the model to maintain the expressive power of Transformers while operating at real-time speeds.
  3. Auxiliary Geometry Features: The model incorporates geometric hints, such as distance from the predicted surface, to help the fusion module prioritize the most reliable camera views for any given point.
The DG-Recon pipeline: 2D features and depth priors are back-projected into a sparse 3D space and fused via linear cross-attention for multi-scale TSDF prediction.

Experimental Results

The system was evaluated on the ScanNet, 7-Scenes, and SUN3D datasets. DG-Recon achieved a state-of-the-art F-score of 0.694 on ScanNet, a substantial improvement over the 0.619 achieved by the previous leading online method, NeuralRecon. Despite the added complexity of depth estimation, the system maintains high performance, running at 20–34 Frames Per Second (FPS).

Qualitative comparison showing how depth-guided fusion (DG-Recon) reduces "ghosting" artifacts and improves surface completeness compared to average-based methods.

Qualitative analysis demonstrates that DG-Recon produces significantly sharper geometric details, such as the distinct edges of furniture, which are often blurred or lost in previous online reconstruction methods. Furthermore, the model exhibits strong generalization capabilities, showing an 18% improvement in F-score on the 7-Scenes dataset without environment-specific fine-tuning, proving that depth guidance makes the model more robust to novel environments.

Conclusion and Future Work

DG-Recon demonstrates that integrating depth priors into the feature fusion process effectively resolves the trade-off between reconstruction quality and computational speed. While the current implementation is highly efficient, future work may explore shared backbones for depth estimation and feature extraction to further reduce latency. Additionally, extending the model's robustness to outdoor environments and extreme camera distortions remains an active area of research.

Summary updated

BibTeX
@inproceedings{ju2023dg,
  author = {Jihong Ju and Ching Wei Tseng and Oleksandr Bailo and Georgi Dikov and Mohsen Ghafoorian},
  title = {{DG-Recon: Depth-Guided Neural 3D Scene Reconstruction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01667},
}
Figure