MVPGS: Excavating Multi-view Priors for Gaussian Splatting from Sparse Input Views

Wangze Xu, Huachen Gao, Shihe Shen, Rui Peng, Jianbo Jiao, Ronggang Wang

Introduction

3D Gaussian Splatting (3DGS) has revolutionized the field of neural rendering by providing high-fidelity, real-time scene reconstruction. By utilizing an explicit point-based representation, 3DGS achieves rendering speeds of hundreds of frames per second, significantly outperforming traditional Neural Radiance Fields (NeRFs). However, this efficiency comes at a cost: 3DGS is notoriously "data-hungry." While it excels when provided with hundreds of images, its performance collapses in sparse-view scenarios—such as when only three images are available.

In these few-shot settings, 3DGS typically suffers from severe overfitting, manifesting as "floaters" or geometry that only appears coherent from the specific training viewpoints. Furthermore, the standard 3DGS pipeline relies on Structure-from-Motion (SfM) tools like COLMAP for initialization; in sparse-view cases, COLMAP often fails to generate any usable point cloud, leaving the model with no foundation to build upon. To address these challenges, the researchers introduce MVPGS (Multi-view Priors for Gaussian Splatting), a framework designed to "excavate" geometric and appearance priors from pre-trained models to stabilize 3DGS in sparse-view environments.

The MVPGS Framework

MVPGS introduces three primary technical innovations to overcome the limitations of sparse-view reconstruction: learning-based initialization, forward-warped appearance priors, and hybrid geometric regularization.

1. MVS-based Initialization

Rather than relying on traditional SfM, which struggles with limited overlap, MVPGS utilizes MVSFormer, a learning-based Multi-view Stereo (MVS) model. By leveraging a model pre-trained on diverse datasets, MVPGS can generate a dense and accurate initial point cloud even from as few as three images. This provides the 3D Gaussians with a robust geometric "head start," ensuring the optimization process begins with a meaningful representation of the scene's structure.

2. Forward Warping Appearance Priors

The core novelty of MVPGS lies in its method for supervising novel views. To prevent the model from overfitting to the sparse training images, the authors generate "pseudo-ground truth" images for unseen camera poses. They achieve this by "warping" pixels from known images into random target poses using the geometry derived from the MVS model.

Unlike previous methods that often use backward warping (fetching colors for a target grid), MVPGS employs forward warping with reversed bilinear sampling. This approach is more naturally compatible with the point-based nature of 3DGS, effectively distributing source pixel colors to target views and providing the necessary supervision to ensure the model learns a consistent 3D representation rather than a set of 2D image-space shortcuts.

3. Hybrid Geometric Regularization

To maintain geometric integrity in areas where visual data is thin, MVPGS incorporates two distinct depth-based losses:

  • Consistent Structure Loss ($L_{CS}$): This loss forces the 3D Gaussians to remain aligned with the reliable depth maps predicted by the MVS model.
  • Monocular Depth Rank Loss ($L_{mono}$): In textureless regions where MVS models might struggle to find correspondences, MVPGS utilizes a pre-trained monocular depth model (DPT). This provides a global sense of depth ordering, helping the model resolve ambiguities in the scene's layout.

Experimental Evaluation

The researchers evaluated MVPGS across four diverse real-world benchmarks: LLFF (forward-facing scenes), DTU (object-centric scans), NVS-RGBD (indoor scenes), and Tanks and Temples (large-scale environments).

Quantitative Performance

In the rigorous 3-view challenge, MVPGS demonstrated state-of-the-art results. On the DTU dataset, it achieved a PSNR of 20.50, a significant improvement over standard 3DGS (14.90) and a notable lead over NeRF-based few-shot competitors like FreeNeRF (19.92).

Efficiency and Speed

One of the most striking advantages of MVPGS is its ability to maintain the real-time rendering benefits of 3DGS. While few-shot NeRF methods often require several seconds to render a single frame (averaging 0.03 to 0.07 FPS), MVPGS achieves rendering speeds of 366 FPS on DTU and 132 FPS on LLFF. This represents a thousand-fold increase in speed without sacrificing visual quality.

Conclusion

MVPGS proves that the "data hunger" of 3D Gaussian Splatting can be mitigated by effectively leveraging external priors. By replacing traditional initialization with MVS-based structures and introducing a novel forward-warping mechanism for appearance supervision, the framework enables high-quality, real-time 3D reconstruction from minimal input.

Currently, the model assumes Lambertian surfaces, meaning it may struggle with non-Lambertian effects like sharp specular reflections or complex lighting changes. Future iterations of the framework may incorporate view-dependent appearance modeling to better capture the nuances of reflective and metallic surfaces. Nevertheless, MVPGS marks a significant step forward in making real-time 3D reconstruction accessible for sparse-view applications.

Summary updated

BibTeX
@inproceedings{xu2024mvpgs,
  author = {Wangze Xu and Huachen Gao and Shihe Shen and Rui Peng and Jianbo Jiao and Ronggang Wang},
  title = {{MVPGS: Excavating Multi-view Priors for Gaussian Splatting from Sparse Input Views}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72970-6\_12},
}
Figure