TL;DR
Mobile-GS is a novel framework that brings high-quality 3D Gaussian Splatting (3DGS) to smartphones in real-time. By eliminating the expensive depth-sorting bottleneck and employing neural compression, it achieves over 100 FPS on mobile hardware and reduces storage requirements from hundreds of megabytes to just 4.8 MB without sacrificing visual fidelity.
What Problem Does This Paper Address?
3D Gaussian Splatting has revolutionized 3D reconstruction, allowing for photorealistic, real-time rendering on high-end desktops. However, moving this technology to mobile devices (like your phone or AR glasses) hits two major walls:
- The Sorting Bottleneck: Standard 3DGS relies on "alpha blending," which requires sorting millions of tiny 3D "blobs" (Gaussians) from front-to-back for every single frame. On mobile GPUs, this sorting process consumes nearly half of the total rendering time.
- Storage Bloat: A single high-quality 3DGS scene can take up gigabytes of space, making it impractical for mobile apps or web-based viewing.
Methods
The authors introduce Mobile-GS, which rethinks the rendering pipeline specifically for edge devices through several key innovations:
- Depth-Aware Order-Independent Rendering: Instead of sorting Gaussians, the authors propose a weighting scheme that allows Gaussians to be blended in any order. They use a learnable function that gives higher weight to closer, larger Gaussians, enabling massive parallelization on the GPU.
- Neural View-Dependent Enhancement: Removing the sorting order usually causes "ghosting" or transparency artifacts. To fix this, they use a lightweight Multi-Layer Perceptron (MLP) that predicts a "correction factor" for opacity based on the camera's viewing angle.
- First-Order SH Distillation: Original 3DGS uses 3rd-order Spherical Harmonics (SH) to model how light changes with the viewing angle, requiring 16 coefficients per color channel. Mobile-GS uses a "teacher-student" distillation approach to compress this down to 1st-order SH (only 4 coefficients), drastically reducing memory.
- Neural Vector Quantization (NVQ): They group Gaussian attributes into clusters and store them using codebooks and Huffman coding, further shrinking the model size.
- Contribution-Based Pruning: The system identifies and removes Gaussians that have low opacity or a very small spatial scale, as these contribute the least to the final image.
Datasets
The researchers evaluated Mobile-GS on three standard, challenging benchmarks for 3D reconstruction:
- Mip-NeRF 360: A dataset featuring complex, unbounded indoor and outdoor scenes.
- Tanks & Temples: High-resolution scans of large-scale environments.
- Deep Blending: Scenes designed to test the limits of view-dependent effects and geometry.
Key Insights and Ideas
- Sorting is the Enemy: The "aha moment" here is the realization that a perfect depth sort isn't strictly necessary for visual quality if you have a smart weighting function and a neural network to "clean up" the resulting artifacts.
- Joint Pruning: Most methods prune based on opacity alone. Mobile-GS shows that considering both opacity and spatial scale is far more effective at removing redundant data.
- Power Efficiency: By removing the sorting step and using compact MLPs, the method doesn't just run faster; it draws significantly less power (0.83W vs. 5.89W for standard 3DGS), which is critical for mobile battery life and thermal management.
Key Results
- Mobile Performance: Achieved 116 FPS at 1600 × 1063 resolution on a Snapdragon 8 Gen 3 mobile GPU.
- Desktop Speed: Reached a blistering 1098 FPS on an RTX 3090 Ti.
- Extreme Compression: Reduced the storage size of the "Bicycle" scene from 839.9 MB (original 3DGS) to just 4.6 MB, a nearly 200x reduction.
- Visual Quality: Maintained a PSNR (image quality metric) of ~27.12, which is nearly identical to the original 3DGS (~27.21) but with a fraction of the computational cost.
Limitations and Future Work
- Training Overhead: While inference (viewing the scene) is lightning-fast on mobile, training the model still requires a powerful desktop GPU and takes significant time.
- Generalization: Like most 3DGS methods, Mobile-GS is "per-scene," meaning you have to train a new model for every new environment you want to capture.
- Quantization Artifacts: In extremely high-texture areas, the aggressive compression can sometimes lead to minor color shifts or slight blurring, a trade-off for the tiny file size.
Summary updated