TL;DR
RLSplat is a new feed-forward 3D Gaussian Splatting (3DGS) framework that achieves "resource-linearity," meaning its inference latency and memory usage scale down linearly with the requested Gaussian budget. By replacing wasteful "predict-then-prune" strategies with a learned allocation policy and an efficient hierarchical decoder, it delivers higher reconstruction quality than existing methods while being up to 4.1× faster and using 9.6× less peak memory at low budgets.
What Problem Does This Paper Address?
3D Gaussian Splatting has revolutionized real-time 3D reconstruction, but deploying it in resource-constrained environments (like mobile AR/VR or robotics) remains challenging. While "feed-forward" methods can reconstruct scenes in a single pass, they often produce a massive, redundant set of Gaussians.
Recent "budget-controllable" methods allow users to specify a target number of Gaussians at inference time. However, these methods have a major efficiency flaw: they typically predict a dense set of candidates first and then prune or select the best ones. This is computationally wasteful—the system still pays the full price in latency and peak memory to generate a representation it immediately throws away. The authors identify this as a lack of inference cost-controllability.
Methods
RLSplat shifts the paradigm from pruning to allocation. Instead of generating too much and cutting back, it learns to generate exactly what is needed.
- Gaussian Allocation Policy: The system uses a geometry backbone to encode images into "anchor tokens," each representing a local 3D region. A policy network then decides how to distribute the total Gaussian budget $B$ across these anchors. Because this allocation is a discrete, non-differentiable decision, the authors train the policy using Group Sequence Policy Optimization (GSPO), a reinforcement learning technique that uses final rendering quality as a reward signal.
- Resource-Linear Gaussian Decoding: Once the budget is allocated, the decoder spawns exactly $B$ Gaussian tokens. To refine these tokens without the $O(B^2)$ cost of global self-attention, RLSplat uses a Hierarchical Neighbor Search. It first finds the nearest anchors and then searches for the nearest Gaussians only within those neighboring anchors. This allows for local cross-attention that scales linearly ($O(B)$) with the budget.
- Locality Regularizers: To ensure the hierarchical search remains accurate, the authors introduce a Voronoi loss and an offset loss. these keep Gaussians physically close to their parent anchors, preventing them from drifting into regions where the neighbor search would fail.
Datasets
- Training: The model is trained on RealEstate10K (RE10K), a large-scale dataset of indoor and outdoor scenes with camera poses.
- Evaluation: The authors perform zero-shot evaluation on the ACID dataset (aerial footage of nature/coastal scenes) to test the model's ability to generalize to unseen environments.
Key Insights and Ideas
- Inference Cost Matters: The core "aha" moment is realizing that true budget control must apply to the process of generation, not just the result. Linear scaling of VRAM and latency is crucial for real-world deployment.
- Allocation > Pruning: By formulating the problem as resource allocation, the model learns to prioritize complex regions (like thin structures or high-texture areas) while spending fewer Gaussians on flat surfaces like walls or water.
- RL for 3D Geometry: Using GSPO to optimize discrete allocations shows that reinforcement learning can effectively bridge the gap between non-differentiable resource decisions and differentiable rendering losses.
Key Results
RLSplat consistently outperforms prior budget-controllable baselines (EcoSplat and F4Splat) across all metrics (PSNR, SSIM, LPIPS) and all budgets.
- Efficiency: As shown in Figure 1, RLSplat's latency and memory decrease linearly, while baselines remain nearly constant. At a low budget of 2K Gaussians, RLSplat is 4.1× faster and uses 9.6× less peak VRAM than F4Splat.
- Quality: At the same 2K budget, it improves PSNR by 0.9 dB over F4Splat. It even outperforms "budget-fixed" methods that use 64× more Gaussians.
- Qualitative Superiority: RLSplat is significantly better at preserving thin, high-frequency structures (like stair balusters or railings) that baselines tend to blur out when the budget is tight.
Limitations and Future Work
- Fixed Encoding Cost: While the decoding and refinement are linear, the initial image encoding by the geometry backbone remains a fixed cost regardless of the budget. This sets a "floor" for how much speedup can be achieved at extremely low budgets.
- Resolution: The current evaluation is limited to a resolution of 256×256. Scaling to higher resolutions while maintaining resource-linearity is a natural next step.
- Downstream Tasks: The authors suggest that RLSplat's compact and cost-controlled representations could be ideal for 3D vision-language models and robotics, which often have their own strict resource limits.
Summary updated