TL;DR
Single-view feed-forward 3D Gaussian Splatting (3DGS) models suffer from severe spatial redundancy because they predict a fixed number of Gaussians for every camera ray. PhGS introduces a plug-and-play, post-hoc framework that prunes redundant Gaussians using a novel importance score and refines the remaining ones with a lightweight recurrent module. It cuts memory usage and Gaussian counts by 50% to 75% with negligible impact on rendering quality, while boosting novel-view rendering speeds by over 56%.
What Problem Does This Paper Address?
Feed-forward 3D Gaussian Splatting has revolutionized 3D reconstruction by generating explicit 3D scenes from images in a single forward pass. However, when dealing with a single input image, networks cannot rely on multi-view consistency to resolve occlusions. To compensate, state-of-the-art single-view models predict a fixed number of Gaussians ($K > 1$) along every single camera ray.
This "K-Gaussians-per-ray" strategy creates massive spatial redundancy. Flat walls or uniform backgrounds end up packed with unnecessary geometric primitives, inflating memory consumption and slowing down rendering. While compaction methods exist for multi-view setups, they are tightly coupled with multi-view architectures and cannot be easily adapted to single-view models. PhGS solves this by introducing a backbone-agnostic approach that operates entirely on the output space of any pre-trained single-view model without requiring retraining.
Methods
The PhGS pipeline consists of two main phases: importance-score-based pruning and recurrent refinement.
- Importance-Score-Based Pruning: Instead of randomly dropping primitives, PhGS computes an importance score for each Gaussian by combining an edge prior from the input image (using a Laplacian operator) with the predicted opacity of the Gaussian. Primitives with high opacity or those aligned with structural boundaries are prioritized. A binary mask is then probabilistically sampled to drop a user-specified fraction of the Gaussians.
- Recurrent Refinement Module: Pruning naturally leaves gaps in the 3D scene. To restore visual fidelity, the remaining Gaussians are passed through a lightweight, two-layer MLP. This module iteratively predicts additive updates to the Gaussians' parameters (opacity, scale, rotation, and position). To stay scalable and avoid expensive spatial operations like kNN or global attention, the MLP processes each Gaussian independently while incorporating a global-context feature via average pooling.
- Keep Ratio Mix (krMix): To prevent having to train a separate refinement network for every desired compression level, the authors introduce a training strategy called krMix. By randomly sampling the target keep ratio during training, a single model learns to dynamically support arbitrary keep ratios (e.g., 25%, 50%, 75%) at inference time.
Datasets
The authors trained and evaluated their method using the following benchmarks:
- RealEstate10K: Used for primary in-domain training and quantitative evaluation.
- KITTI & DL3DV: Used to evaluate the zero-shot cross-domain generalization capabilities of the refinement module on outdoor driving scenes and diverse real-world environments, respectively.
Key Insights and Ideas
- Decoupling is Feature-Rich: By treating the underlying feed-forward network as a frozen black box, PhGS avoids the massive computational burden of retraining heavy base models. It can be applied retroactively to any architecture (e.g., Flash3D, Niagara, SHARP).
- "Inflating" Surviving Primitives: Statistical analysis reveals that the recurrent refinement module compensates for missing geometry primarily by shifting the opacity and scaling distributions of the surviving Gaussians to the right. Essentially, the remaining Gaussians become slightly larger and more opaque to fill the gaps left by pruned neighbors.
- Asymmetric Generalization: The authors discovered that a refinement module trained at a fixed 50% keep ratio generalizes well when asked to prune more aggressively (25%), but fails dramatically when pruning less aggressively (75%). The krMix strategy successfully mitigates this issue, ensuring stable performance across all ratios.
Key Results
The authors evaluated PhGS across three distinct backbones on the RealEstate10K dataset at a 50% keep ratio ($r=0.5$):
- Fidelity Preservation: For Flash3D, the PSNR dropped by a mere 0.06 dB while cutting the Gaussian count in half. For SHARP, which produces an exceptionally dense native point cloud, PhGS actually improved the PSNR by 0.30 dB while reducing storage from 66.06 MB to 33.03 MB.
- Throughput Boost: On an NVIDIA RTX 4070 Ti, compressing Flash3D reconstructions by 50% increased the novel-view rendering throughput from 1,443 views per second to 2,257 views per second—a 56.4% speedup.
- Baselines Outperformed: PhGS consistently outperformed existing multi-view post-hoc pruning methods like LightGaussian and PUP 3D-GS adapted to the single-view setting, demonstrating the value of its edge-opacity scoring function.
Limitations and Future Work
While PhGS significantly accelerates downstream novel-view rendering, the initial full-inference pipeline experiences a 35% runtime overhead due to the iterative unrolling of the recurrent refinement module (adding roughly 76 ms per scene).
Additionally, because the recurrent module only receives error feedback from the single source view during inference, its ability to correct geometric artifacts that only become visible from extreme, highly oblique novel viewpoints remains limited. Future work could explore incorporating self-supervised geometric priors to better guide the refinement process from unseen angles.
Summary updated