TL;DR
Vanilla 3D Gaussian Splatting (3DGS) inherits NeRF’s single-view-per-iteration training paradigm, which frequently causes Gaussian primitives to overfit to individual viewpoints and produce blurry artifacts or floaters. MVGS introduces a plug-and-play multi-view regulation framework combining multi-view gradient updates, coarse-to-fine intrinsic scaling, and cross-ray 3D densification. Across static, dynamic, reflective, and large-scale scenes, MVGS consistently boosts novel-view rendering quality by roughly 1 dB PSNR while producing more compact 3D representations.
What Problem Does This Paper Address?
Traditional 3D Gaussian Splatting optimizes scene representations by rasterizing Gaussians from one camera viewpoint at a time. While this approach matches NeRF’s training format, 3DGS uses an explicit set of 3D kernels rather than a continuous implicit MLP. Optimizing these explicit primitives against only a single view per step frequently leads them to overfit to specific camera angles.
This single-view bias results in geometry distortion, "floater" artifacts, and degraded appearance when synthesizing novel views—particularly under challenging conditions like specularity, mirror reflections, thin structures, or dynamic scene motion.
Methods
MVGS replaces the single-view update rule with a multi-view supervision and densification pipeline that can be dropped into existing 3DGS codebases in fewer than 200 lines of code:
[ Multi-Resolution Image Pyramid ]
│
▼
[ Multi-View Regulated Learning (MVRL) ] ──> Joint gradients across M viewpoints
│
▼
[ Cross-Ray Densification (CRD) ] ──> Cast rays from 2D loss peaks;
densify Gaussians in 3D intersections
│
▼
[ Multi-View Augmented Densification ] ──> Dynamically lower split thresholds
when viewpoint disparities are large
- Multi-View Regulated Learning (MVRL): Instead of calculating loss on a single image, MVGS samples $M$ views per step and accumulates their gradients before backpropagating. This constrains Gaussian attributes (covariance, opacity, position) to jointly satisfy multiple vantage points.
- Cross-Intrinsic Guidance (CIG): To avoid memory bottlenecks and maintain global-to-local consistency, training follows a 4-level image pyramid (downsampling by 8×, 4×, 2×, down to 1×). Early coarse stages ingest wide view batches (e.g., $M = 48$) to lock in coarse geometry, while later stages refine high-frequency details with fewer views ($M = 8$).
- Cross-Ray Densification (CRD): High-error patches are identified on 2D loss maps using sliding windows. The model casts rays from these window vertices into 3D space, identifies ray intersection volumes (cuboids) across camera views, and prioritizes splitting or cloning Gaussians situated in these shared regions.
- Multi-View Augmented Densification (MVAD): When sampled training views have large baseline translations (measured on a normalized unit sphere), MVGS adaptively halves the densification gradient threshold $\beta$, promoting aggressive primitive splitting where perspective differences are widest.
Datasets
The authors evaluate MVGS across several benchmarks without introducing new datasets:
- Static 3D Scenes: Mip-NeRF 360, Tanks & Temples, and Deep Blending.
- Reflective Objects: Shiny Blender and Glossy Synthetic datasets.
- Dynamic (4D) Reconstruction: D-NeRF synthetic dynamic benchmark.
- Large-Scale / Multi-Scale Scenes: BungeeNeRF dataset.
Key Insights and Ideas
- Explicit representations make multi-view gradients viable: NeRF struggles with multi-view iterations because coordinates sampled along rays rarely intersect across cameras. In contrast, 3DGS kernels have fixed 3D parameterizations, meaning the same physical Gaussian directly receives blended gradient signals from multiple perspectives simultaneously.
- Training longer does not fix single-view overfitting: Extending training iterations on baseline 3DGS plateaus well below MVGS performance. The failure mode of baseline 3DGS is structural (lack of multi-view constraints per step), not a lack of optimization steps.
- More views are not always better: Performance peaks at a moderate view batch size (around 9 to 18 views depending on scene type). Ingesting too many views per step over-constrains local regions and hurts reconstruction.
Key Results
- General 3D Reconstruction: On Tanks & Temples, adding MVGS to vanilla 3DGS improves PSNR from 23.14 dB to 24.44 dB (+1.30 dB). Integrating it into Scaffold-GS raises performance from 23.96 dB to 25.54 dB (+1.58 dB).
- Reflective Scenes: Applied to 3DGS-DR, MVGS achieves state-of-the-art results on Shiny Blender (34.61 dB vs. 34.08 dB) and Glossy Synthetic (30.81 dB vs. 30.13 dB), producing sharper specular highlights and environment reflections.
- 4D Dynamic Rendering: Combined with 4DGS on the D-NeRF dataset, average PSNR increases from 34.05 dB to 35.11 dB (+1.06 dB). Rendering speed improves from 82 FPS to 102 FPS, while model storage decreases from 18 MB to 12 MB due to more compact primitive placement.
Limitations and Future Work
- Computational and Memory Overhead: Processing multiple views per step demands higher peak GPU VRAM during training, requiring the authors to rely on resolution downscaling in early stages.
- Batch Size Tuning: The optimal view batch size varies by scene configuration, and over-saturating the batch size can degrade rendering quality.
- Future Work: Not explicitly discussed by the authors.
Summary updated