TL;DR
NeRFPrior introduces a novel approach to 3D surface reconstruction by using a fast-trained Neural Radiance Field (NeRF) as a geometric and photometric prior to guide the learning of a Signed Distance Function (SDF). By leveraging the NeRF's density and color outputs, the method enforces multi-view visibility and depth constraints, achieving state-of-the-art indoor scene reconstruction without relying on large-scale, pre-trained datasets.
What Problem Does This Paper Address?
Reconstructing high-quality 3D surfaces from a set of 2D images is a classic computer vision problem. Recently, neural implicit functions (like NeuS) have become the go-to method, but they struggle to accurately reconstruct complex geometries and textureless regions (like blank walls or floors) using only multi-view images.
To fix this, recent methods rely on priors—extra hints about the scene's geometry. However, current approaches have major flaws:
- Data-driven priors (e.g., depth or normal maps generated by models pre-trained on massive datasets) often fail to generalize to out-of-distribution scenes.
- Overfitting priors (e.g., sparse point clouds generated by traditional Multi-View Stereo algorithms like COLMAP) only provide sparse geometric clues and completely lack color (photometric) information, which is vital for accurate volume rendering.
The authors tackle this gap by asking: Can we extract a dense, scene-specific prior that contains both geometry and color, without needing external training data?
Methods
The authors propose a two-stage pipeline that uses a fast-converging NeRF as a teacher for an SDF network:
- The NeRF Prior: First, they train a grid-based NeRF (specifically TensoRF) on the target scene. Because grid-based NeRFs are highly optimized, this takes only about 30 minutes. This frozen NeRF now acts as a continuous 3D field providing density (geometry) and view-dependent color.
- SDF Optimization with Prior Supervision: They then train an SDF network. For points sampled along a camera ray, they query the frozen NeRF for density and color to directly supervise the SDF network's predictions.
- Occlusion-Aware Multi-View Consistency: To refine the surface, the network finds where a ray intersects the surface and checks if that point is visible from other camera views. NeRFPrior uses the NeRF's view-dependent color to perform a highly accurate "local-prior volume rendering" visibility check. If the point is visible, the network enforces photometric consistency across the views.
- Depth Consistency Loss: Multi-view consistency fails on textureless surfaces. To handle this, the authors introduce a heuristic: if a region has low color variance (textureless) and low density variance in the NeRF prior (flat plane), the network applies a depth consistency loss. This forces neighboring points to have similar depths, resulting in smooth walls and floors.
Key Results
The authors evaluated their method on three standard indoor scene benchmarks: ScanNet, BlendSwap, and Replica.
- Superior Accuracy: On the Replica dataset, NeRFPrior achieved a Chamfer Distance of 0.038, significantly outperforming baselines like NeuS (0.066) and MonoSDF (0.075).
- State-of-the-Art on Real Data: On the real-world ScanNet dataset, it achieved the highest F1 score (0.782 when integrated with monocular cues), capturing fine details like bedside lamps that other methods blurred out.
- Faster Training: Because the NeRF prior guides the SDF network to converge quickly, the total pipeline takes about 4.7 hours. This is notably faster than competing methods like NeuS (7.2 hours) and MonoSDF (10.6 hours).
Key Insights
- NeRFs are excellent, unbiased priors: Instead of relying on a depth-estimation model trained on millions of unrelated images, a NeRF trained on the exact same scene provides a highly accurate, unbiased prior that perfectly matches the target distribution.
- Color is crucial for visibility checks: Because NeRFs natively model view-dependent color, they can be used to perform vastly superior occlusion and visibility checks compared to traditional MVS.
- Dynamic smoothing for textureless areas: By combining color variance and density variance, the model can dynamically detect flat, textureless walls and apply smoothing only where it makes sense, preserving sharp details on complex objects.
Limitations and Future Work
While highly effective, the initial NeRF density field can be noisy. To work around this, the authors manually schedule multi-view and depth consistency losses to activate only after 100k–150k training iterations.
Additionally, the pipeline requires training two separate networks sequentially. Future work could explore end-to-end training paradigms or attempt to adapt these continuous SDF extraction techniques to newer, faster representations like 3D Gaussian Splatting.
Summary updated