Self-Evolving Depth-Supervised 3D Gaussian Splatting from Rendered Stereo Pairs

Sadra Safadoust, Fabio Tosi, Fatma Güney, Matteo Poggi

Abstract

While 3D Gaussian Splatting (3DGS) has achieved state-of-the-art performance in real-time photorealistic rendering, it often suffers from poor underlying geometric reconstruction. The optimization process, driven primarily by photometric loss, frequently introduces "floaters" and noisy surfaces that degrade depth accuracy. This paper presents a self-evolving framework that leverages the model's own rendering capabilities to improve its geometric integrity. By generating virtual stereo pairs and processing them through a pre-trained stereo matching network, the system produces high-quality depth priors that supervise the 3DGS model, leading to significantly cleaner geometry and improved performance in sparse-view scenarios.

Introduction

3D Gaussian Splatting (3DGS) has emerged as a powerful alternative to Neural Radiance Fields (NeRFs), offering comparable visual quality with significantly faster rendering speeds. However, the structural accuracy of 3DGS often lags behind its visual fidelity. Because the optimization focuses on minimizing pixel-wise color differences, the model may place transparent or misplaced Gaussians—commonly referred to as "floaters"—to account for complex lighting or textures.

This lack of geometric consistency is a significant hurdle for downstream applications such as robotics, augmented reality (AR), and 3D reconstruction. Existing Depth-Supervised (DS-GS) methods attempt to mitigate this by using external cues like Structure-from-Motion (SfM) points or monocular depth estimation. However, SfM points are often too sparse, and monocular depth is frequently scale-ambiguous and fails to generalize across diverse environments. This work introduces a "self-evolving" pipeline that allows the model to refine its own geometry using stereo vision principles.

Methodology

The proposed Self-Evolving DS-GS pipeline creates a feedback loop where the model acts as its own teacher. The process consists of several key stages:

1. Initial Optimization

The model undergoes an initial training phase for several thousand iterations. This "warm-up" period ensures the model can render reasonably stable images, even if the underlying 3D structure remains noisy.

2. Virtual Stereo Rendering

For any given training view, the framework renders a second "virtual" view by applying a small horizontal shift to the camera position. This creates a rectified stereo pair. The insight here is that while the 3D Gaussians may be disorganized, the 2D renders they produce are often geometrically consistent enough to be interpreted by stereo vision algorithms.

3. Stereo Supervision and Disparity Estimation

The rendered stereo pairs are processed by a pre-trained deep stereo matching network, such as RAFT-Stereo. The network predicts a high-quality disparity map from the rendered images. This disparity is then converted into a depth map, which serves as a pseudo-ground-truth prior to supervise the 3DGS model.

4. Self-Evolution and Caching

To maintain training efficiency, the system does not run the stereo network at every iteration. Instead, it caches the predicted disparity maps and refreshes them periodically. As the 3DGS model improves its rendering quality, the stereo network provides increasingly accurate depth priors, creating a self-evolving cycle of geometric refinement.

Experimental Results

The authors benchmarked the method against three diverse datasets: ETH3D (real-world high-resolution), ScanNet++ (complex indoor environments), and BlendedMVS (semi-synthetic).

Quantitative Performance

The method demonstrates a substantial leap in depth accuracy. On the ETH3D dataset, the self-evolving approach achieved an Absolute Relative (Abs Rel) error of 0.057. This is a marked improvement over vanilla 3DGS (0.211) and outperforms other supervision methods such as SfM-based priors (0.109) and monocular depth priors (0.153). Furthermore, the method showed improved image synthesis metrics (PSNR and SSIM), particularly in challenging sparse-view settings where standard 3DGS typically fails to maintain coherence.

Qualitative Improvements

Visual analysis of the rendered depth maps reveals a significant reduction in artifacts. The "foggy" noise and floaters prevalent in standard 3DGS are largely eliminated, resulting in sharp, well-defined surfaces that accurately reflect the scene's true structure.

Discussion and Conclusion

A key insight of this research is the "hallucination advantage": the fact that 2D renders often contain latent geometric truths that the 3D model has not yet fully realized. By using binocular stereo rather than Multi-View Stereo (MVS), the authors minimize computational overhead while maximizing generalization across different scene types.

While the method is highly effective, it remains dependent on a successful initial training phase; if the initial renders are too poor, the stereo network cannot provide meaningful feedback. Future work may involve extending this framework to dynamic scenes or integrating it with foundation models for even more robust geometry. Ultimately, this paper demonstrates that by looking at itself from a slightly different perspective, a 3D Gaussian Splatting model can effectively teach itself the structural integrity required for high-fidelity 3D reconstruction.

Summary updated

Abstract

Abstract
BibTeX
@inproceedings{safadoust2024self,
  author = {Sadra Safadoust and Fabio Tosi and Fatma Güney and Matteo Poggi},
  title = {{Self-Evolving Depth-Supervised 3D Gaussian Splatting from Rendered Stereo Pairs}},
  booktitle = {BMVC},
  year = {2024},
  url = {https://openreview.net/forum?id=mclh4M2RgD},
}
Figure