Indoor Scene Reconstruction with Fine-Grained Details Using Hybrid Representation and Normal Prior Enhancement

Sheng Ye, Yubin Hu, Matthieu Lin, Yu-Hui Wen, Wang Zhao, Yong-Jin Liu, Wenping Wang

TL;DR

This paper introduces a novel neural implicit framework for high-fidelity 3D indoor scene reconstruction from multi-view RGB images. By decoupling the scene into a hybrid representation (MLPs for smooth areas and tri-planes for fine details) and introducing an uncertainty-aware normal prior enhancement module, the method successfully captures both large flat walls and intricate high-frequency geometries.

What Problem Does This Paper Address?

Reconstructing 3D indoor scenes from a sequence of 2D RGB images is a foundational challenge in computer vision with widespread applications in robotics, virtual reality, and gaming. Recent state-of-the-art methods leverage neural implicit functions—specifically Signed Distance Fields (SDFs) optimized via volume rendering—combined with monocular geometric priors (like surface normals) to guide the reconstruction.

However, indoor scenes present a difficult dual challenge: they contain large, textureless, flat areas (floors, walls) alongside highly detailed, complex structures (furniture, tabletop items). Standard Multi-Layer Perceptrons (MLPs) suffer from a "smoothness bias," meaning they excel at flat surfaces but smooth out fine details. Conversely, voxel-grid alternatives can capture details but introduce noise and consume massive amounts of memory. Furthermore, off-the-shelf monocular normal estimators often produce noisy or erroneous predictions around thin, intricate structures, misleading the network during training.

Methods

The authors address these limitations through two core innovations:

  1. Hybrid Geometry Representation: Instead of relying on a single large MLP, the architecture splits the scene into two parallel branches. An MLP branch captures the rough, low-frequency outline of the scene. Concurrently, a memory-efficient tri-plane representation (three axis-aligned orthogonal feature planes) captures high-frequency geometric complements. A shallow decoder regresses these tri-plane features into residual SDF values. Crucially, the MLP is initialized to represent a bounding sphere while the tri-plane is initialized to zero, allowing the network to implicitly decouple smooth macro-structures from sharp micro-structures without requiring explicit semantic labels.
  2. Prior Enhancement & Uncertainty Estimation:
  3. Image Preprocessing: Input images undergo 2D sharpening (via a $3 \times 3$ convolutional kernel) and denoising (median filtering) before normal estimation, which significantly improves the initial monocular normal predictions.
  4. Uncertainty Estimation Module: A U-Net architecture takes the RGB images, estimated normals, and high-level structure-aware features from a visual foundation model (DINO) to predict a pixel-wise uncertainty map. This uncertainty acts as a dynamic weight in the normal loss function: regions with high uncertainty (such as complex geometry where monocular estimators fail) rely more heavily on realistic RGB color loss, preventing inaccurate normal priors from degrading fine details.

Datasets

The method was evaluated on two primary benchmarks, alongside real-world captures:

  • ScanNet: A real-world indoor dataset captured with Kinect V1 RGB-D cameras, characterized by significant real-world noise, motion blur, and sensor artifacts.
  • Replica: A high-quality synthetic dataset featuring dense, clean geometry and annotations.
  • Real-world Mobile Captures: To demonstrate practical generalization, the authors collected 2-minute hand-held video sequences using mobile phones, utilizing OpenMVG to estimate camera poses.
Fig. 1. Our proposed method can reconstruct fine and accurate indoor scenes only from a sequence of posed RGB images.

Key Insights and Ideas

  • Implicit Decoupling via Initialization: By carefully initializing the MLP branch to a bounding sphere and the tri-plane branch to zero, the network naturally learns to allocate smooth macro-structures to the MLP and sharp micro-structures to the tri-plane, bypassing the need for expensive semantic annotations.
  • 2D Image Processing Impacts 3D Reconstruction: Simple 2D image enhancement (sharpening and denoising) directly translates to cleaner 3D geometry because it improves the fidelity of the monocular normal priors fed into the volume renderer.
  • Foundation Models for Geometric Uncertainty: Utilizing DINO features (which naturally capture pixel-level structural information during self-supervised learning) proves superior to language-aligned models like CLIP for predicting where geometric priors are likely to fail.

Key Results

  • Quantitative Performance: On ScanNet, the proposed method achieved a reconstruction F-score of 0.773, outperforming state-of-the-art baselines like MonoSDF (0.733) and HelixSurf (0.755). On the Replica dataset, it achieved a dominant F-score of 0.936 (compared to MonoSDF's 0.915).
  • Qualitative Superiority: Reconstructed meshes show significantly sharper edges on furniture, clearer separation of small objects on tables, and fewer bumpy artifacts on flat walls compared to voxel- or pure MLP-based approaches.
  • Resource Efficiency: Despite the hybrid architecture, the model is highly memory-efficient, consuming less memory (11GB) during training than NeuralAngelo (21GB) or MonoSDF (16GB).

Limitations and Future Work

  • Specular and Transparent Surfaces: Because the method relies on standard neural radiance fields and volume rendering, it fails to accurately reconstruct mirrors or glass surfaces due to physical reflections and refractions. The authors suggest integrating Whitted or Monte Carlo ray tracing to solve this.
  • Training Speed: The optimization process takes 1.5 to 2 hours per scene, leaving room for future work in real-time or accelerated training frameworks.
  • Scale: Extending this fine-grained reconstruction capability to very large-scale, multi-room environments remains an open challenge.

Summary updated

Abstract

The reconstruction of indoor scenes from multi-view RGB images is challenging due to the coexistence of flat and texture-less regions alongside delicate and fine-grained regions. Recent methods leverage neural radiance fields aided by predicted surface normal priors to recover the scene geometry. These methods excel in producing complete and smooth results for floor and wall areas. However, they struggle to capture complex surfaces with high-frequency structures due to the inadequate neural representation and the inaccurately predicted normal priors. This work aims to reconstruct high-fidelity surfaces with fine-grained details by addressing the above limitations. To improve the capacity of the implicit representation, we propose a hybrid architecture to represent low-frequency and high-frequency regions separately. To enhance the normal priors, we introduce a simple yet effective image sharpening and denoising technique, coupled with a network that estimates the pixel-wise uncertainty of the predicted surface normal vectors. Identifying such uncertainty can prevent our model from being misled by unreliable surface normal supervisions that hinder the accurate reconstruction of intricate geometries. Experiments on the benchmark datasets show that our method outperforms existing methods in terms of reconstruction quality. Furthermore, the proposed method also generalizes well to real-world indoor scenarios captured by our hand-held mobile phones. Our code is publicly available at: https://github.com/yec22/Fine-Grained-Indoor-Recon.
BibTeX
@article{ye2023indoor,
  author = {Sheng Ye and Yubin Hu and Matthieu Lin and Yu-Hui Wen and Wang Zhao and Yong-Jin Liu and Wenping Wang},
  title = {{Indoor Scene Reconstruction with Fine-Grained Details Using Hybrid Representation and Normal Prior Enhancement}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2023},
  url = {https://openreview.net/forum?id=o_ZlcwmYAL},
}
Figure