MVGS: Multi-view-regulated Gaussian Splatting for Novel View Synthesis

Xiaobiao Du, Yida Wang, Xin YuSubmission to ICLR 2025 · Withdrawn

TL;DR

Vanilla 3D Gaussian Splatting (3DGS) inherits NeRF’s single-view-per-iteration training paradigm, which frequently causes Gaussian primitives to overfit to individual viewpoints and produce blurry artifacts or floaters. MVGS introduces a plug-and-play multi-view regulation framework combining multi-view gradient updates, coarse-to-fine intrinsic scaling, and cross-ray 3D densification. Across static, dynamic, reflective, and large-scale scenes, MVGS consistently boosts novel-view rendering quality by roughly 1 dB PSNR while producing more compact 3D representations.

What Problem Does This Paper Address?

Traditional 3D Gaussian Splatting optimizes scene representations by rasterizing Gaussians from one camera viewpoint at a time. While this approach matches NeRF’s training format, 3DGS uses an explicit set of 3D kernels rather than a continuous implicit MLP. Optimizing these explicit primitives against only a single view per step frequently leads them to overfit to specific camera angles.

This single-view bias results in geometry distortion, "floater" artifacts, and degraded appearance when synthesizing novel views—particularly under challenging conditions like specularity, mirror reflections, thin structures, or dynamic scene motion.

Methods

MVGS replaces the single-view update rule with a multi-view supervision and densification pipeline that can be dropped into existing 3DGS codebases in fewer than 200 lines of code:

[ Multi-Resolution Image Pyramid ]
           │
           ▼
[ Multi-View Regulated Learning (MVRL) ] ──> Joint gradients across M viewpoints
           │
           ▼
[ Cross-Ray Densification (CRD) ]         ──> Cast rays from 2D loss peaks;
                                              densify Gaussians in 3D intersections
           │
           ▼
[ Multi-View Augmented Densification ]    ──> Dynamically lower split thresholds
                                              when viewpoint disparities are large
  • Multi-View Regulated Learning (MVRL): Instead of calculating loss on a single image, MVGS samples $M$ views per step and accumulates their gradients before backpropagating. This constrains Gaussian attributes (covariance, opacity, position) to jointly satisfy multiple vantage points.
  • Cross-Intrinsic Guidance (CIG): To avoid memory bottlenecks and maintain global-to-local consistency, training follows a 4-level image pyramid (downsampling by 8×, 4×, 2×, down to 1×). Early coarse stages ingest wide view batches (e.g., $M = 48$) to lock in coarse geometry, while later stages refine high-frequency details with fewer views ($M = 8$).
  • Cross-Ray Densification (CRD): High-error patches are identified on 2D loss maps using sliding windows. The model casts rays from these window vertices into 3D space, identifies ray intersection volumes (cuboids) across camera views, and prioritizes splitting or cloning Gaussians situated in these shared regions.
  • Multi-View Augmented Densification (MVAD): When sampled training views have large baseline translations (measured on a normalized unit sphere), MVGS adaptively halves the densification gradient threshold $\beta$, promoting aggressive primitive splitting where perspective differences are widest.

Datasets

The authors evaluate MVGS across several benchmarks without introducing new datasets:

  • Static 3D Scenes: Mip-NeRF 360, Tanks & Temples, and Deep Blending.
  • Reflective Objects: Shiny Blender and Glossy Synthetic datasets.
  • Dynamic (4D) Reconstruction: D-NeRF synthetic dynamic benchmark.
  • Large-Scale / Multi-Scale Scenes: BungeeNeRF dataset.

Key Insights and Ideas

  • Explicit representations make multi-view gradients viable: NeRF struggles with multi-view iterations because coordinates sampled along rays rarely intersect across cameras. In contrast, 3DGS kernels have fixed 3D parameterizations, meaning the same physical Gaussian directly receives blended gradient signals from multiple perspectives simultaneously.
  • Training longer does not fix single-view overfitting: Extending training iterations on baseline 3DGS plateaus well below MVGS performance. The failure mode of baseline 3DGS is structural (lack of multi-view constraints per step), not a lack of optimization steps.
  • More views are not always better: Performance peaks at a moderate view batch size (around 9 to 18 views depending on scene type). Ingesting too many views per step over-constrains local regions and hurts reconstruction.

Key Results

  • General 3D Reconstruction: On Tanks & Temples, adding MVGS to vanilla 3DGS improves PSNR from 23.14 dB to 24.44 dB (+1.30 dB). Integrating it into Scaffold-GS raises performance from 23.96 dB to 25.54 dB (+1.58 dB).
  • Reflective Scenes: Applied to 3DGS-DR, MVGS achieves state-of-the-art results on Shiny Blender (34.61 dB vs. 34.08 dB) and Glossy Synthetic (30.81 dB vs. 30.13 dB), producing sharper specular highlights and environment reflections.
  • 4D Dynamic Rendering: Combined with 4DGS on the D-NeRF dataset, average PSNR increases from 34.05 dB to 35.11 dB (+1.06 dB). Rendering speed improves from 82 FPS to 102 FPS, while model storage decreases from 18 MB to 12 MB due to more compact primitive placement.

Limitations and Future Work

  • Computational and Memory Overhead: Processing multiple views per step demands higher peak GPU VRAM during training, requiring the authors to rely on resolution downscaling in early stages.
  • Batch Size Tuning: The optimal view batch size varies by scene configuration, and over-saturating the batch size can degrade rendering quality.
  • Future Work: Not explicitly discussed by the authors.

Summary updated

Abstract

Recent works in volume rendering, \textit{e.g.} NeRF and 3D Gaussian Splatting (3DGS), significantly advance the rendering quality and efficiency with the help of the learned implicit neural radiance field or 3D Gaussians. Rendering on top of an explicit representation, the vanilla 3DGS and its variants deliver real-time efficiency by optimizing the parametric model with single-view supervision per iteration during training which is adopted from NeRF. Consequently, certain views are overfitted, leading to unsatisfying appearance in novel-view synthesis and imprecise 3D geometries. To solve aforementioned problems, we propose a new 3DGS optimization method embodying four key novel contributions: 1) We transform the conventional single-view training paradigm into a multi-view training strategy. With our proposed multi-view regulation, 3D Gaussian attributes are further optimized without overfitting certain training views. As a general solution, we improve the overall accuracy in a variety of scenarios and different Gaussian variants. 2) Inspired by the benefit introduced by additional views, we further propose a cross-intrinsic guidance scheme, leading to a coarse-to-fine training procedure concerning different resolutions. 3) Built on top of our multi-view regulated training, we further propose a cross-ray densification strategy, densifying more Gaussian kernels in the ray-intersect regions from a selection of views. 4) By further investigating the densification strategy, we found that the effect of densification should be enhanced when certain views are distinct dramatically. As a solution, we propose a novel multi-view augmented densification strategy, where 3D Gaussians are encouraged to get densified to a sufficient number accordingly, resulting in improved reconstruction accuracy. We conduct extensive experiments to demonstrate that our proposed method is capable of improving novel view synthesis of the Gaussian-based explicit representation methods about 1 dB PSNR for various tasks. \href{https://mvgs666.github.io/}{\textcolor{magenta}{Codes are available.}}
BibTeX
@misc{du2024mvgs,
  author = {Xiaobiao Du and Yida Wang and Xin Yu},
  title = {{MVGS: Multi-view-regulated Gaussian Splatting for Novel View Synthesis}},
  year = {2024},
  eprint = {2410.02103},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.02103},
}
Figure