TL;DR
PreSem-Surf is a new NeRF-based optimization method that reconstructs high-quality 3D surfaces from RGB-D sequences by integrating semantic information and a novel "pre-rendering" sampling mechanism. By using a coarse-to-fine progressive modeling strategy, it achieves a superior balance between reconstruction smoothness and geometric accuracy while significantly improving training efficiency.
What Problem Does This Paper Address?
While Neural Radiance Fields (NeRF) have revolutionized 3D scene representation, applying them to surface reconstruction from RGB-D (color + depth) data remains challenging. Existing methods often face a "tug-of-war" between two issues:
- Resource Consumption: Standard MLPs are computationally expensive and slow to train.
- Reconstruction Quality: Low-quality or noisy input data often leads to blurred models, artifacts, or "bloated" geometries where fine details (like chair legs or window frames) are lost.
Traditional voxel-based or hash-table approaches speed things up but can struggle to distinguish between sensor noise and actual local details. This paper aims to bridge that gap by teaching the model what it is looking at (semantics) and where to focus its efforts (progressive sampling).
Methods
The authors introduce two primary innovations to the standard NeRF pipeline:
1. SG-MLP Pre-Rendering Mechanism
Instead of jumping straight into full-color rendering, the model uses a Sampling-Guided Multi-Layer Perceptron (SG-MLP).
- PR-MLP: A "Preconditioning" MLP first estimates the initial volume density of the scene using uniform sampling.
- Dynamic Thresholding: The system calculates a dynamic threshold based on the mean and maximum density of the previous layer. This allows the model to ignore empty space and perform "importance sampling" only in regions where surfaces actually exist.
2. Progressive Semantic Modeling (PFPSMS)
The authors argue that humans understand scenes hierarchically—first the layout, then the objects. Their Progressive Semantic Modeling Strategy mimics this:
- Pseudo-color Guidance: Since many datasets lack semantic labels, they use DFormer to generate semantic "pseudo-color" images.
- Coarse-to-Fine Rendering: In the first half of training, the model uses voxels 10 times larger than the final size to capture the global structure. In the second half, it switches to fine-grained rendering to capture intricate details. This prevents the model from getting stuck in "local optima" (essentially getting distracted by small details before the big picture is clear).
Datasets
The model was evaluated primarily on:
- Synthetic Dataset: 7 different scenes where noise and artifacts were intentionally added to simulate real-world depth sensors.
- ScanNet: Used specifically for parameter analysis in real-world scenarios (Scenes 0000 and 0012).
- NYU40 Dataset: Used as the label mapping standard for the semantic categories.
Key Insights and Ideas
- Semantics as a Regularizer: Semantic information isn't just for labeling; it acts as a powerful prior for geometry. Knowing a group of pixels belongs to a "chair" helps the model maintain structural consistency even when depth data is noisy.
- The "Scout" Approach: By using a simplified MLP (the PR-MLP) to "scout" the scene's density first, the model avoids the massive computational waste of querying complex networks for empty air.
- Voxel Scaling: Increasing voxel size by 10x during the initial phase acts as a natural low-pass filter, ensuring the "skeleton" of the room is correct before the "skin" (fine details) is applied.
Key Results
PreSem-Surf was benchmarked against state-of-the-art models like Co-slam, Neural RGBD, and GO-Surf.
- Quantitative Superiority: On average across all scenes, PreSem-Surf achieved the highest scores in C-L1 (0.0236), F-score (0.9440), and IoU (0.6389).
- Qualitative Improvements: Visual comparisons show that while competitors like GO-Surf often produce "bloated" or fragmented reconstructions (e.g., gaps in chair backs), PreSem-Surf maintains sharp edges and smooth surfaces.
- Ablation Success: Removing the semantic module or the SG-MLP resulted in a significant drop in performance, proving that both the "what" (semantics) and the "where" (guided sampling) are critical to the model's success.
Limitations and Future Work
The authors identify one major hurdle: Scalability. Because the method relies on a voxel-based framework, memory usage and time costs increase rapidly as the scene scale grows. While it performs excellently on individual rooms (like the "Morning Apartment"), reconstructing entire buildings or large outdoor areas would currently require prohibitive amounts of VRAM. The authors suggest that further optimizing the underlying data structure is the primary direction for future research.
Summary updated