Abstract
Sparse2DGS is a novel 3D surface reconstruction method that generates highly accurate 3D meshes from just three input images. By combining a deep learning stereo model (DUSt3R) with traditional Multi-View Stereo (COLMAP) to create a dense initial point cloud, it overcomes the poor initialization problem that usually plagues Gaussian Splatting in sparse-view scenarios. Sparse2DGS achieves state-of-the-art accuracy in under two minutes, significantly outperforming existing NeRF-based methods in efficiency while maintaining comparable geometric precision.
Introduction
Reconstructing a 3D object from 2D images—known as Multi-View Stereo (MVS)—traditionally requires dozens or hundreds of overlapping photos. While newer techniques like Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) have revolutionized novel view synthesis, they still struggle when given only a handful of images (e.g., three).
For Gaussian Splatting specifically, the optimization process relies heavily on an initial sparse point cloud generated by Structure from Motion (SfM). When only three images are available, SfM produces very few 3D points. Without enough starting points, the Gaussians cannot properly cover the object, resulting in failed optimizations and poor 3D geometry. While some NeRF-based methods can handle sparse views, they often require massive datasets and days of pre-training. Sparse2DGS addresses this by achieving fast, accurate 3D reconstruction from just three images without extensive pre-training.
Methods
The core novelty of Sparse2DGS lies in its initialization pipeline, which builds upon 2D Gaussian Splatting (2DGS). Unlike standard 3DGS, 2DGS uses flat 2D ellipses, making it better suited for extracting solid surfaces.
The pipeline consists of several key steps:
- Dense Point Cloud Generation: The system feeds three input images into DUSt3R (a foundation model for stereo vision) and COLMAP MVS.
- Point Cloud Integration: DUSt3R provides dense geometry but may have coordinate errors, while COLMAP offers high metric accuracy but sparse coverage. The authors align the DUSt3R cloud with the COLMAP cloud using the Iterative Closest Point (ICP) algorithm.
- Gaussian Optimization: This fused, dense point cloud serves as the starting position for 2D Gaussians. These are optimized using color loss, depth distortion loss, and normal consistency loss.
- No Densification: Because the initial point cloud is already dense, the standard Gaussian densification step (cloning/splitting) is skipped, significantly speeding up optimization.
- Mesh Extraction: Depth maps rendered from the optimized Gaussians are fused into a 3D mesh using Truncated Signed Distance Function (TSDF) integration.
Experimental Setup
The method was evaluated on the DTU dataset, using 15 scenes resized to 800x600. To simulate a challenging sparse-view scenario, the authors selected 3-image subsets with minimal overlap. Scans containing highly reflective objects (scans 69, 97, and 110) were excluded, as specular highlights break the multi-view consistency assumptions required for accurate reconstruction.
Key Results
Sparse2DGS demonstrates a significant leap in both accuracy and efficiency for sparse-view reconstruction.
- Accuracy: Sparse2DGS achieved a mean Chamfer Distance of 1.51 on the DTU dataset, tying for first place with UFORecon, a state-of-the-art NeRF-based method.
- Speed: On a single RTX 4090 GPU, optimization takes approximately 100 seconds, and mesh reconstruction takes 10 seconds. This is a massive improvement over UFORecon, which requires days of pre-training to achieve similar results.
- Ablation Study: The custom initialization is critical. Using standard SfM initialization resulted in a poor Chamfer Distance of 2.54, whereas the fused DUSt3R + COLMAP initialization reduced the error to 1.51.
Discussion and Conclusion
The primary insight of this work is that the failure of Gaussian Splatting in sparse views is primarily an initialization bottleneck, not a limitation of the renderer. By combining the dense estimation capabilities of DUSt3R with the metric precision of COLMAP, Sparse2DGS provides the necessary geometric anchors for stable optimization. Furthermore, skipping the densification phase makes the process incredibly fast.
Despite its success, the method struggles with highly reflective surfaces. Additionally, the current pipeline relies on ground-truth camera parameters. Future work aims to jointly optimize camera poses and incorporate view-dependent effects to handle complex materials and uncalibrated image sets.
Summary updated