TL;DR
RadSplat bridges the gap between high-quality Neural Radiance Fields (NeRFs) and high-speed 3D Gaussian Splatting (3DGS). By using a NeRF as a robust "teacher" to supervise a point-based representation, RadSplat achieves state-of-the-art visual quality while rendering at over 900 FPS—outperforming 3DGS in both speed and robustness.
What Problem Does This Paper Address?
In the world of 3D scene reconstruction, there has long been a trade-off between quality and speed. Neural Radiance Fields (NeRFs) produce stunning, photorealistic results and handle "in-the-wild" challenges like changing exposure or motion blur very well. However, they are notoriously slow because they require "volumetric rendering"—sampling hundreds of points along every single camera ray.
On the other hand, 3D Gaussian Splatting (3DGS) is incredibly fast because it uses a rasterization-based approach (similar to traditional computer graphics). But 3DGS is "brittle." Its optimization often fails on complex real-world scenes, leading to "floater" artifacts or blurry results when lighting varies between photos. Furthermore, 3DGS models can become bloated with millions of points, slowing down performance on mobile devices or large scenes.
Methods
RadSplat introduces a three-stage pipeline to get the robustness of NeRFs with the speed of splatting:
- NeRF-Informed Initialization & Supervision: Instead of training on raw, "noisy" images (which might have different exposures), the authors first train a robust NeRF (Zip-NeRF). They then use this NeRF to initialize the positions of the Gaussians and, more importantly, to provide "clean" supervision. The 3DGS model learns from the NeRF's idealized renders rather than the inconsistent original photos.
- Ray Contribution-Based Pruning: Standard 3DGS uses simple heuristics to decide which points to keep. RadSplat introduces a novel pruning technique that calculates the actual contribution of each Gaussian to the final image. If a Gaussian doesn't significantly help the visual quality of any training view, it’s deleted.
- Viewpoint-Based Visibility Filtering: For large scenes (like an entire house), rendering every Gaussian in the scene for every frame is wasteful. RadSplat clusters camera positions and pre-calculates a "visibility list" for each cluster. At test time, it only renders the Gaussians relevant to your current location.
Datasets
The authors evaluated RadSplat on two primary benchmarks:
- Mip-NeRF 360: The standard benchmark for unbounded 3D scenes, containing 9 indoor and outdoor environments.
- Zip-NeRF Dataset: A collection of 4 large-scale scenes (entire apartments and houses) that include challenging lighting and exposure variations.
Key Insights and Ideas
- The NeRF as a "Clean" Teacher: One of the biggest "aha" moments is that raw data is often too messy for point-based optimization. By using a NeRF to "factor out" lighting and exposure noise, the resulting Gaussians are much more stable and accurate.
- Efficiency through Pruning: The authors found that they could match the quality of standard 3DGS while using 10x fewer Gaussians. This massive reduction in point count is the primary driver for their 900+ FPS rendering speeds.
- Max vs. Mean: When calculating importance for pruning, using the maximum contribution across views is better than the average. This ensures that points representing small details visible from only one angle aren't accidentally deleted.
Key Results
RadSplat sets a new bar for the speed-quality Pareto front:
- Speed: It renders at 907 FPS on a standard RTX 3090 GPU, which is 3.6x faster than 3DGS and a staggering 3,000x faster than Zip-NeRF.
- Quality: Despite the speed, it achieves higher SSIM (Structural Similarity) scores than Zip-NeRF on several benchmarks, meaning the images are more faithful to the original scene.
- Compactness: The "Ours Light" model uses only ~0.37 million Gaussians compared to 3DGS's 3.16 million for the same scene, making it much easier to store and stream.
Limitations and Future Work
While RadSplat outperforms other real-time methods, the authors note there is still a small quality gap compared to the full (non-real-time) Zip-NeRF on extremely large-scale scenes. Future work may involve further refining how visibility lists are generated or exploring even more aggressive compression techniques to bring these high-fidelity scenes to lower-end mobile hardware.
Summary updated