FewViewGS: Gaussian Splatting with Few View Matching and Multi-stage Training

Ruihong Yin, Vladimir Yugay, Yue Li, Sezer Karaoglu, Theo Gevers

Introduction

3D Gaussian Splatting (3DGS) has emerged as a transformative technique for novel view synthesis (NVS), offering high-fidelity reconstructions and real-time rendering speeds. However, the performance of 3DGS is heavily dependent on dense input data. When limited to a "few-shot" scenario—typically defined as three or fewer training images—the unstructured nature of the Gaussians leads to severe overfitting. In these sparse-view settings, the model often "memorizes" the training views, resulting in significant artifacts such as "floaters," blurry regions, and broken geometry when viewed from novel perspectives.

Existing attempts to mitigate these issues often rely on integrating massive, pre-trained generative priors, such as depth estimators or image diffusion models. While these can provide helpful guidance, they introduce significant computational overhead, increase training times, and are prone to "domain shift" errors when the target scene differs from the prior's training data. FewViewGS addresses these challenges by prioritizing geometric consistency derived from 2D image matching over external generative priors.

Methodology

FewViewGS introduces a framework that enforces consistency across novel views through a multi-stage training pipeline and robust pixel-level correspondences. The approach is built upon three primary pillars:

1. Multi-stage Training Scheme

The training process is divided into three distinct phases to ensure stable convergence and high-quality geometry:

  • Pre-training Stage: The model is initially optimized using only the available sparse training views. This establishes a baseline scene structure and provides initial depth maps for subsequent stages.
  • Intermediate Stage: This is the core of the FewViewGS framework. The system samples virtual "novel" viewpoints between the known training frames. Using a robust image matcher (RoMa), the system identifies correspondences between training images, warps these pixels into the novel viewpoint, and enforces color, geometry, and semantic consistency.
  • Tuning Stage: A final refinement phase focuses on the original known views to sharpen fine details and finalize the reconstruction.

2. Matching-based Consistency

Rather than relying on a depth network to predict scene structure, FewViewGS utilizes explicit 2D pixel correspondences. By finding matching points between two training images $i$ and $j$, the system "warps" these points to a virtual camera $k$ positioned between them. By ensuring the rendered view at camera $k$ aligns with these warped points, the model is forced to learn a more accurate 3D structure that satisfies the geometric constraints of multiple views.

To handle potential inaccuracies in matching or occlusions, the authors employ a "Min" trick during supervision. When supervising a novel view from two different training images, the system applies a min() operation on the loss, allowing the model to ignore outlier matches that might otherwise corrupt the scene geometry.

3. Locality Preserving Regularization

Sparse reconstructions often suffer from a "noisy" appearance. To counteract this, FewViewGS introduces a regularization loss that encourages neighboring Gaussians in 3D space to share similar color properties. This acts as a smoothing prior, effectively removing rendering artifacts while preserving the underlying local structure of the scene.

Experimental Evaluation

The framework was evaluated on three standard benchmarks for novel view synthesis: the DTU Dataset (object-centric), the LLFF Dataset (forward-facing real-world scenes), and the Synthetic Blender Dataset. The primary testing configuration focused on the highly challenging 3-view setup.

Key Results

  • State-of-the-Art Accuracy: On the LLFF 3-view benchmark, FewViewGS achieved a PSNR of 20.54, outperforming both the original 3DGS (19.48) and specialized few-shot methods like FSGS (20.43).
  • Computational Efficiency: FewViewGS demonstrates a significant advantage in training speed. It requires approximately 5.8 minutes on a single NVIDIA A6000, whereas competing methods like SparseGS and FSGS require 51.7 minutes and 10.9 minutes, respectively.
  • Semantic Consistency: By utilizing a pre-trained VGG16 network for semantic supervision, the model maintains high-level scene consistency even in areas where pixel-perfect matches are difficult to obtain.

Discussion and Conclusion

The success of FewViewGS suggests that explicit 2D correspondences can provide stronger and more reliable geometric constraints for 3DGS than modern "black-box" depth estimation networks. By leveraging classical geometric principles within a modern Gaussian Splatting framework, the authors achieve a balance of speed and fidelity that was previously difficult to reach in sparse-view settings.

While the method excels at geometric reconstruction, it still faces challenges in perfectly rendering texture-rich regions when the novel viewpoint is significantly distant from the training views. Future work may involve integrating denser feature-matching networks or extending the framework to handle dynamic scenes. Nonetheless, FewViewGS represents a significant step forward in making 3D Gaussian Splatting viable for real-world applications where only limited data is available.

Summary updated

BibTeX
@inproceedings{yin2024fewviewgs,
  author = {Ruihong Yin and Vladimir Yugay and Yue Li and Sezer Karaoglu and Theo Gevers},
  title = {{FewViewGS: Gaussian Splatting with Few View Matching and Multi-stage Training}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=liHe9iumIi},
}
Figure