Abstract
We present a novel approach for recovering 3D shape and view dependent appearance from a few colored images, enabling efficient 3D reconstruction and novel view synthesis. Our method learns an implicit neural representation in the form of a Signed Distance Function (SDF) and a radiance field. The model is trained progressively through ray marching enabled volumetric rendering, and regularized with learning-free multi-view stereo (MVS) cues. Key to our contribution is a novel implicit neural shape function learning strategy that encourages our SDF field to be as linear as possible near the level-set, hence robustifying the training against noise emanating from the supervision and regularization signals. Without using any pretrained priors, our method, called SparseCraft, achieves state-of-the-art performances both in novel-view synthesis and reconstruction from sparse views in standard benchmarks, while requiring less than 10 minutes for training.
Introduction
Reconstructing a 3D scene from images is a foundational challenge in computer vision. While Neural Radiance Fields (NeRF) have revolutionized the field, they typically require dense image sets. In "sparse-view" scenarios—where only three to six images are available—most neural methods struggle, often producing "floater" artifacts or incomplete geometry.
Existing solutions frequently rely on expensive pre-trained priors learned from massive 3D datasets. However, these priors can fail when encountering out-of-distribution scenes. SparseCraft addresses this by bridging the gap between traditional geometric matching and modern implicit neural representations, achieving high-quality reconstruction "from scratch" without external priors.
Methodology
SparseCraft represents a scene using a Signed Distance Function (SDF) for shape and a Radiance Field for appearance. The architecture integrates three primary innovations to handle sparse data efficiently:
- Stereopsis Guided Geometric Linearization: To prevent the network from overfitting to noise in the initial MVS point cloud, the authors enforce a Taylor expansion-based loss. This forces the SDF to remain as linear as possible near the surface, acting as a geometric denoiser that favors smooth, simple surfaces.
- MVS Cue Integration: The model leverages traditional Multi-View Stereo (COLMAP) to provide not just positional anchors, but also surface normals and color labels to regularize training.
- Fast Progressive Learning: Utilizing multi-resolution hash encoding, the model "warms up" by starting with coarse details and gradually activating higher resolutions, preventing fine-grained noise from disrupting early training stages.
Experiments and Results
The model was evaluated on the DTU Dataset, BlendedMVS, and Tanks & Temples. SparseCraft demonstrates significant improvements in both geometry and image quality:
- Reconstruction Accuracy: On the DTU benchmark, SparseCraft achieved a mean Chamfer Distance of 1.04, outperforming generalizable models like VolRecon and ReTR that require extensive pre-training.
- Synthesis Quality: The method significantly outperformed state-of-the-art sparse-view methods like RegNeRF and FreeNeRF in PSNR and SSIM metrics, particularly in extreme 3-view scenarios.
- Efficiency: By avoiding heavy pre-training and using optimized data structures, SparseCraft reaches peak performance in under 10 minutes on a single GPU.
Discussion and Conclusion
The core philosophy of SparseCraft is "Occam’s Razor for Geometry": in sparse-view scenarios, the simplest mathematical function that fits the data is often the most accurate. By forcing the SDF to be linear near the surface, the model avoids the complex, noisy geometry that typically plagues few-shot reconstruction.
While the method currently relies on COLMAP for initialization—making it sensitive to cases with extremely low image overlap—it demonstrates that traditional photogrammetry and neural implicit fields are highly complementary. Future work may involve replacing static MVS cues with learnable components to further increase robustness to challenging lighting and materials.
Summary updated