TL;DR
This paper introduces a novel neural implicit framework for high-fidelity 3D indoor scene reconstruction from multi-view RGB images. By decoupling the scene into a hybrid representation (MLPs for smooth areas and tri-planes for fine details) and introducing an uncertainty-aware normal prior enhancement module, the method successfully captures both large flat walls and intricate high-frequency geometries.
What Problem Does This Paper Address?
Reconstructing 3D indoor scenes from a sequence of 2D RGB images is a foundational challenge in computer vision with widespread applications in robotics, virtual reality, and gaming. Recent state-of-the-art methods leverage neural implicit functions—specifically Signed Distance Fields (SDFs) optimized via volume rendering—combined with monocular geometric priors (like surface normals) to guide the reconstruction.
However, indoor scenes present a difficult dual challenge: they contain large, textureless, flat areas (floors, walls) alongside highly detailed, complex structures (furniture, tabletop items). Standard Multi-Layer Perceptrons (MLPs) suffer from a "smoothness bias," meaning they excel at flat surfaces but smooth out fine details. Conversely, voxel-grid alternatives can capture details but introduce noise and consume massive amounts of memory. Furthermore, off-the-shelf monocular normal estimators often produce noisy or erroneous predictions around thin, intricate structures, misleading the network during training.
Methods
The authors address these limitations through two core innovations:
- Hybrid Geometry Representation: Instead of relying on a single large MLP, the architecture splits the scene into two parallel branches. An MLP branch captures the rough, low-frequency outline of the scene. Concurrently, a memory-efficient tri-plane representation (three axis-aligned orthogonal feature planes) captures high-frequency geometric complements. A shallow decoder regresses these tri-plane features into residual SDF values. Crucially, the MLP is initialized to represent a bounding sphere while the tri-plane is initialized to zero, allowing the network to implicitly decouple smooth macro-structures from sharp micro-structures without requiring explicit semantic labels.
- Prior Enhancement & Uncertainty Estimation:
- Image Preprocessing: Input images undergo 2D sharpening (via a $3 \times 3$ convolutional kernel) and denoising (median filtering) before normal estimation, which significantly improves the initial monocular normal predictions.
- Uncertainty Estimation Module: A U-Net architecture takes the RGB images, estimated normals, and high-level structure-aware features from a visual foundation model (DINO) to predict a pixel-wise uncertainty map. This uncertainty acts as a dynamic weight in the normal loss function: regions with high uncertainty (such as complex geometry where monocular estimators fail) rely more heavily on realistic RGB color loss, preventing inaccurate normal priors from degrading fine details.
Datasets
The method was evaluated on two primary benchmarks, alongside real-world captures:
- ScanNet: A real-world indoor dataset captured with Kinect V1 RGB-D cameras, characterized by significant real-world noise, motion blur, and sensor artifacts.
- Replica: A high-quality synthetic dataset featuring dense, clean geometry and annotations.
- Real-world Mobile Captures: To demonstrate practical generalization, the authors collected 2-minute hand-held video sequences using mobile phones, utilizing OpenMVG to estimate camera poses.
Key Insights and Ideas
- Implicit Decoupling via Initialization: By carefully initializing the MLP branch to a bounding sphere and the tri-plane branch to zero, the network naturally learns to allocate smooth macro-structures to the MLP and sharp micro-structures to the tri-plane, bypassing the need for expensive semantic annotations.
- 2D Image Processing Impacts 3D Reconstruction: Simple 2D image enhancement (sharpening and denoising) directly translates to cleaner 3D geometry because it improves the fidelity of the monocular normal priors fed into the volume renderer.
- Foundation Models for Geometric Uncertainty: Utilizing DINO features (which naturally capture pixel-level structural information during self-supervised learning) proves superior to language-aligned models like CLIP for predicting where geometric priors are likely to fail.
Key Results
- Quantitative Performance: On ScanNet, the proposed method achieved a reconstruction F-score of 0.773, outperforming state-of-the-art baselines like MonoSDF (0.733) and HelixSurf (0.755). On the Replica dataset, it achieved a dominant F-score of 0.936 (compared to MonoSDF's 0.915).
- Qualitative Superiority: Reconstructed meshes show significantly sharper edges on furniture, clearer separation of small objects on tables, and fewer bumpy artifacts on flat walls compared to voxel- or pure MLP-based approaches.
- Resource Efficiency: Despite the hybrid architecture, the model is highly memory-efficient, consuming less memory (11GB) during training than NeuralAngelo (21GB) or MonoSDF (16GB).
Limitations and Future Work
- Specular and Transparent Surfaces: Because the method relies on standard neural radiance fields and volume rendering, it fails to accurately reconstruct mirrors or glass surfaces due to physical reflections and refractions. The authors suggest integrating Whitted or Monte Carlo ray tracing to solve this.
- Training Speed: The optimization process takes 1.5 to 2 hours per scene, leaving room for future work in real-time or accelerated training frameworks.
- Scale: Extending this fine-grained reconstruction capability to very large-scale, multi-room environments remains an open challenge.
Summary updated