RTGS: Enabling Real-Time Gaussian Splatting on Mobile Devices Using Efficiency-Guided Pruning and Foveated Rendering

Weikai Lin, Yu Feng, Yuhao ZhuarXiv 2024 · arXiv · OpenReview

Abstract

Point-Based Neural Rendering (PBNR), specifically 3D Gaussian Splatting (3DGS), has emerged as a promising technique for real-time, photorealistic rendering in AR/VR and digital twins. However, achieving real-time performance on mobile devices remains a significant challenge. This paper introduces RTGS (also referred to as MetaSapiens), a system that achieves real-time neural rendering on mobile hardware for the first time. RTGS combines efficiency-aware pruning to optimize rendering speed with a novel foveated rendering method that leverages the human visual system's low acuity in peripheral regions. Our system achieves over 100 FPS on an Nvidia Jetson Xavier board without sacrificing subjective visual quality, as validated by user studies. The code is open-sourced at [https://github.com/horizon-research/Fov-3DGS].

Introduction: The Challenge of Mobile PBNR

3D Gaussian Splatting (3DGS) represents scenes as millions of 3D ellipsoids (splats) that are projected and rasterized. While highly efficient on desktop GPUs, 3DGS performance drops significantly on mobile hardware, often falling below 10 FPS. Existing optimization techniques typically focus on pruning points to reduce storage, but this does not necessarily translate to faster rendering. RTGS addresses this by focusing on computational efficiency and human perception.

Key Insight: The Intersection Bottleneck

A critical observation in this work is that rendering latency is not linearly proportional to the total number of points. Instead, the primary computational cost stems from tile-ellipse intersections. A single large Gaussian ellipsoid overlapping many screen tiles is far more expensive to process than numerous smaller ones.

Latency correlates more strongly with the number of tile-ellipse intersections (triangles) than with raw point count (circles).

As shown in the figure above, the number of intersections provides a much more accurate predictor of latency than point count alone. This insight drives the pruning strategy in RTGS, targeting "expensive" points that cover many tiles but contribute little detail.

Methodology

RTGS employs a three-pronged strategy to enable real-time performance on mobile devices:

1. Efficiency-Aware Pruning & Scale Decay

Rather than pruning based solely on a point's contribution to image quality, RTGS defines Computational Efficiency (CE) as the ratio of a point's visual contribution to its compute cost (intersections). Points with low CE are pruned. Furthermore, "Scale Decay" is used to shrink large, computationally expensive ellipsoids, reducing their tile footprint.

2. Foveated PBNR

RTGS introduces the first foveated rendering (FR) method specifically designed for point-based rendering. It exploits the fact that human peripheral vision is less sensitive to detail.

The foveated rendering approach relaxes quality in high-eccentricity regions by matching feature statistics between reference and quality-relaxed images.

The system uses a hierarchical point representation where peripheral regions are rendered using a strict subset of the points used for the fovea. To maintain perceived quality, RTGS utilizes an eccentricity-aware Human Visual System Quality (HVSQ) metric during training and Selective Multi-Versioning to store optimized parameters (like Opacity and DC color) for different quality levels.

3. Hardware Accelerator Co-design

To handle the load imbalance inherent in 3DGS, the authors propose a custom hardware architecture featuring:

  • Tile Merging: Dynamically combining tiles with light workloads to maximize processing unit utilization.
  • Incremental Pipelining: Using line-buffers to allow rasterization to begin before sorting is fully complete, reducing pipeline stalls.

Results and Evaluation

RTGS was evaluated on standard benchmarks including Mip-NeRF 360, Tanks & Temples, and Deep Blending. Key results include:

  • Performance: RTGS-H is 1.9x faster than existing mobile baselines on a Volta GPU. With the custom hardware accelerator, it achieves a 20.9x geomean speedup (up to 27.7x) over the GPU baseline.
  • Visual Quality: A user study with 12 participants using a Meta Quest Pro confirmed that RTGS's foveated rendering is statistically indistinguishable from dense models.
  • Efficiency: Pruning reduced model size by 85% with minimal quality loss, and the hardware design achieved a 56.8x reduction in energy consumption compared to mobile GPUs.

Conclusion

RTGS demonstrates that real-time, high-quality neural rendering is achievable on mobile devices by shifting the focus from raw point counts to computational efficiency and perceptual importance. Future work will explore extending these techniques to dynamic scenes and further optimizing training overhead.

Summary updated

Abstract

Point-Based Neural Rendering (PBNR), i.e., the 3D Gaussian Splatting-family algorithms, emerges as a promising class of rendering techniques, which are permeating all aspects of society, driven by a growing demand for real-time, photorealistic rendering in AR/VR and digital twins. Achieving real-time PBNR on mobile devices is challenging. This paper proposes RTGS, a PBNR system that for the first time delivers real-time neural rendering on mobile devices while maintaining human visual quality. RTGS combines two techniques. First, we present an efficiency-aware pruning technique to optimize rendering speed. Second, we introduce a Foveated Rendering (FR) method for PBNR, leveraging humans' low visual acuity in peripheral regions to relax rendering quality and improve rendering speed. Our system executes in real-time (above 100 FPS) on Nvidia Jetson Xavier board without sacrificing subjective visual quality, as confirmed by a user study. The code is open-sourced at [https://github.com/horizon-research/Fov-3DGS].
BibTeX
@misc{lin2024rtgs,
  author = {Weikai Lin and Yu Feng and Yuhao Zhu},
  title = {{RTGS: Enabling Real-Time Gaussian Splatting on Mobile Devices Using Efficiency-Guided Pruning and Foveated Rendering}},
  year = {2024},
  eprint = {2407.00435},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.00435},
}
Figure