Improving Neural Indoor Surface Reconstruction with Mask-Guided Adaptive Consistency Constraints

Xinyi Yu, Liqin Lu, Jintao Rong, Guangkai Xu, Linlin Ou

Abstract

3D scene reconstruction from 2D images has been a long-standing task. Instead of estimating per-frame depth maps and fusing them in 3D, recent research leverages the neural implicit surface as a unified representation for 3D reconstruction. Equipped with data-driven pre-trained geometric cues, these methods have demonstrated promising performance. However, inaccurate prior estimation, which is usually inevitable, can lead to suboptimal reconstruction quality, particularly in some geometrically complex regions. In this paper, we propose a two-stage training process, decouple view-dependent and view-independent colors, and leverage two novel consistency constraints to enhance detail reconstruction performance without requiring extra priors. Additionally, we introduce an essential mask scheme to adaptively influence the selection of supervision constraints, thereby improving performance in a self-supervised paradigm. Experiments on synthetic and real-world datasets show the capability of reducing the interference from prior estimation errors and achieving high-quality scene reconstruction with rich geometric details.

Introduction

Reconstructing a 3D scene from a set of 2D images is a cornerstone of computer vision, essential for applications ranging from autonomous navigation to virtual reality. Recently, neural implicit surfaces have emerged as the gold standard for this task. However, indoor environments pose significant challenges due to large texture-less surfaces (like white walls) and complex furniture geometries.

Existing methods often rely on "priors"—depth or surface normal estimations from pre-trained models—to guide the reconstruction. Unfortunately, these guesses are frequently inaccurate. If a model treats an incorrect prior as absolute truth, the resulting 3D mesh becomes distorted or loses fine details. This research introduces a framework that adaptively decides when to trust these priors and when to rely on the inherent multi-view consistency of the images themselves.

Methodology

The proposed framework employs a two-stage training process and several key technical innovations to improve reconstruction fidelity.

Two-Stage Training and Color Decoupling

The training begins with a "warm-up" stage using standard RGB reconstruction and estimated normal priors to establish a coarse initial geometry. In the second stage, the model transitions to an "Adaptive Consistency" phase, refining details by comparing its own rendered views. To facilitate this, the system decouples color into two components: view-dependent (specular highlights) and view-independent (the actual base color). This allows the model to enforce a Photometric Consistency Constraint on the base color across different viewpoints, which is significantly more stable for optimization.

Mask-Guided Adaptive Consistency

The core innovation is a mask-guided scheme that utilizes "virtual rays" to enforce consistency in a self-supervised manner.

The overall pipeline of the proposed method, illustrating the use of virtual rays and the mask-guided constraint selection.

The system generates virtual viewpoints and applies three specific masks to guide learning:

  1. Sample Mask: Ensures the virtual viewpoint is actually outside the object.
  2. Occlusion Mask: Identifies if a point is hidden from a certain view to avoid "ghosting" artifacts.
  3. Adaptive Check Mask: Compares rendered normals from different views. If they do not match, the model assumes the initial prior was wrong and switches from a normal prior loss to a geometric consistency loss.

Key Insights: Consistency as a Sanity Check

A fundamental idea of this work is using multi-view consistency as a filter for external priors. Instead of trying to build a "better" prior, the authors built a better way to ignore bad ones. By enforcing both geometric (shape) and photometric (color) consistency, the model is constrained from multiple angles, leading to sharper edges and smoother flat surfaces.

Illustration of Geometric and Photometric Consistency Losses using sampled and virtual viewpoints.

By generating virtual rays during training, the model "hallucinates" what the scene should look like from angles it hasn't seen yet, forcing the underlying 3D representation to be more robust and globally coherent.

Experimental Results

The method was evaluated on the ScanNet (real-world RGB-D) and Replica (high-quality synthetic) datasets, consistently outperforming state-of-the-art baselines like MonoSDF and NeuRIS:

  • Accuracy on ScanNet: The model achieved a leading F-score of 0.780, compared to MonoSDF (0.733) and NeuRIS (0.692).
  • Precision: It reached a precision of 0.837, indicating that the reconstructed surfaces are highly reliable and free of the "floaters" common in other neural methods.
  • Qualitative Quality: Visually, the method excels at recovering thin structures, such as chair legs, and maintaining the planarity of walls that other models often turn into "wavy" surfaces due to noisy depth priors.

Conclusion and Future Work

While the paper demonstrates impressive results, the two-stage training process and the generation of virtual rays add computational complexity compared to simpler one-stage methods. Future research may explore ways to reduce this overhead and extend the approach to dynamic scenes where objects or lighting conditions change over time.

Summary updated

Abstract

3D scene reconstruction from 2D images has been a long-standing task. Instead of estimating per-frame depth maps and fusing them in 3D, recent research leverages the neural implicit surface as a unified representation for 3D reconstruction. Equipped with data-driven pre-trained geometric cues, these methods have demonstrated promising performance. However, inaccurate prior estimation, which is usually inevitable, can lead to suboptimal reconstruction quality, particularly in some geometrically complex regions. In this paper, we propose a two-stage training process, decouple view-dependent and view-independent colors, and leverage two novel consistency constraints to enhance detail reconstruction performance without requiring extra priors. Additionally, we introduce an essential mask scheme to adaptively influence the selection of supervision constraints, thereby improving performance in a self-supervised paradigm. Experiments on synthetic and real-world datasets show the capability of reducing the interference from prior estimation errors and achieving high-quality scene reconstruction with rich geometric details.
BibTeX
@inproceedings{yu2024improving,
  author = {Xinyi Yu and Liqin Lu and Jintao Rong and Guangkai Xu and Linlin Ou},
  title = {{Improving Neural Indoor Surface Reconstruction with Mask-Guided Adaptive Consistency Constraints}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=iSKMMYRL6q},
}
Figure