BudgetSplat: Budget-Controllable Generalizable Gaussian Splatting

Yuxi Hu, Jun Zhang, Wei Mao, Yuchen Rao, Friedrich FraundorferarXiv 2026 · arXiv

TL;DR

BudgetSplat is a feed-forward 3D Gaussian Splatting (3D-GS) framework that allows users to specify a "preservation ratio" (budget) at test time without any retraining. By treating compaction as a joint problem of importance, spatial coverage, and cross-view redundancy, it reduces Gaussian counts by over 60% with negligible impact on rendering quality.

What Problem Does This Paper Address?

Generalizable 3D Gaussian Splatting has revolutionized novel view synthesis by allowing models to reconstruct scenes from a few images in a single feed-forward pass. However, most current methods are "pixel-aligned," meaning they predict one or more Gaussian primitives for every single pixel in the input views.

This leads to a massive scalability issue: as you increase image resolution or the number of input views, the number of Gaussians—and thus the storage and rendering costs—explodes. Furthermore, many of these Gaussians are redundant, observing the same 3D points from different angles. Existing solutions are often inflexible (fixed-capacity) or slow (requiring per-scene optimization). BudgetSplat aims to provide a "knob" that lets users dial in the desired representation size at inference time while maintaining high-fidelity reconstructions.

Methods

The authors propose a three-pronged approach to achieve budget-controllable compaction:

  1. Budget-Conditioned Feature Modulation (BCFM): Instead of just filtering Gaussians after they are predicted, BudgetSplat injects the user-specified preservation ratio ($\rho$) directly into the feature space. A lightweight embedding network transforms this ratio into a feature map that modulates the base Gaussian features. This allows the MLP "head" to adapt Gaussian opacity and attributes specifically for the requested budget.
  2. Intra-Cross Mask Supervision (ICMS): During training, the model needs to learn which Gaussians are "important." ICMS provides this guidance by creating pseudo-labels that prioritize high-frequency structures (edges/details) while ensuring sparse coverage in low-frequency (smooth) regions. Crucially, it uses a voxel-based weighting scheme to down-weight redundant observations across different views, teaching the model to avoid "double-counting" the same 3D space.
  3. Coverage-Aware Budget Selection (CABS): At inference, the model doesn't just pick the top-K Gaussians globally, which might leave some areas empty. Instead, it performs patch-wise pre-selection to ensure spatial distribution, followed by view-wise ranking and 3D non-maximum suppression (NMS) to prune geometrically redundant primitives.
Overview of the BudgetSplat framework, showing the pipeline from input views to budget-adapted compact Gaussians.

Datasets

The model was trained and evaluated on:

  • ScanNet: A large-scale dataset of indoor scenes. The authors used a split with 100 training scenes and 8 test scenes.
  • Replica: Used specifically for zero-shot generalization testing. The model was evaluated on 8 Replica scenes without any fine-tuning to see how well the budget-control logic transfers to unseen environments.

Key Insights and Ideas

  • Budgeting is a Joint Problem: The authors argue that compaction isn't just about "importance ranking." You must balance importance with spatial coverage (don't leave holes) and redundancy (don't repeat the same point).
  • Adaptation vs. Truncation: Simply cutting off the bottom 75% of Gaussians from a standard model (truncation) performs poorly. By making the model aware of the budget during the prediction phase (adaptation), the remaining Gaussians can "stretch" their attributes to compensate for the missing ones.
  • Pseudo-Masks as Teachers: Since there is no ground truth for "Gaussian importance," using image gradients and voxel-grids to create supervision masks is a clever way to bake structural and redundancy awareness into the training process.

Key Results

BudgetSplat demonstrates a robust trade-off between representation size and quality.

  • Efficiency: On ScanNet with 10 context views, BudgetSplat$_{25\%}$ (using only 25% of the budget) achieves rendering quality nearly identical to the full model, while using ~45% fewer Gaussians than the state-of-the-art FreeSplat.
  • Aggressive Compaction: Even at a tiny 5% budget (roughly 33k Gaussians), the model maintains usable quality, outperforming other budgeted methods like EcoSplat.
  • Generalization: In zero-shot tests on the Replica dataset, the model maintained stable performance even when removing ~74% of Gaussians, proving the learned selection strategy isn't just memorizing ScanNet patterns.
Figure 2: Qualitative results on ScanNet [6] with 10 context views. Ours is rendered using only 25% of the Gaussian budget.

Limitations and Future Work

  • Dependency on Heuristics: The pseudo-supervision relies on image gradients and estimated geometry. If the initial depth estimation is poor or the image is degraded, the selection process may fail.
  • Backbone Overhead: While the output representation is compact, the inference pipeline still uses a dense pixel-aligned backbone. The authors suggest that future work could explore budget-aware candidate generation to reduce the computational load of the backbone itself.
  • Refinement: Future iterations could explore more robust supervision methods that don't rely as heavily on simple image gradients.

Summary updated

Abstract

Feed-forward 3D Gaussian Splatting enables efficient generalizable novel view synthesis, but existing pixel-aligned methods often generate highly redundant Gaussian primitives, causing the final representation size and rendering cost to grow rapidly with image resolution and the number of input views. We propose BudgetSplat, a budget-controllable feed-forward framework that adapts the final Gaussian representation to a user-specified preservation ratio. Rather than treating compaction as simple importance ranking, we formulate it as a joint problem of importance estimation, spatial coverage, and cross-view redundancy. A single trained model supports varying numbers of context views and multiple test-time budgets without per-budget retraining or per-scene compression. To realize this formulation, Intra-Cross Mask Supervision provides structure-, coverage-, and redundancy-aware guidance for learning Gaussian importance, while budget-conditioned feature modulation adapts Gaussian opacity and attributes to the requested preservation ratio. Coverage-Aware Budget Selection further combines patch-wise pre-selection, view-wise opacity ranking, and 3D redundancy suppression to construct compact yet spatially distributed Gaussian representations. Experiments on indoor reconstruction and cross-dataset generalization demonstrate consistent quality--size trade-offs across different input view counts and preservation ratios, substantially reducing final representation storage and rendering cost while preserving rendering quality.
BibTeX
@misc{hu2026budgetsplat,
  author = {Yuxi Hu and Jun Zhang and Wei Mao and Yuchen Rao and Friedrich Fraundorfer},
  title = {{BudgetSplat: Budget-Controllable Generalizable Gaussian Splatting}},
  year = {2026},
  eprint = {2611.06593},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06593},
}
Figure