Abstract
Photorealistic reconstruction relying on 3D Gaussian Splatting has shown promising potential in various domains. However, the current 3D Gaussian Splatting system only supports radiance field reconstruction using undistorted perspective images. In this paper, we present OmniGS, a novel omnidirectional Gaussian splatting system, to take advantage of omnidirectional images for fast radiance field reconstruction. Specifically, we conduct a theoretical analysis of spherical camera model derivatives in 3D Gaussian Splatting. According to the derivatives, we then implement a new GPU-accelerated omnidirectional rasterizer that directly splats 3D Gaussians onto the equirectangular screen space for omnidirectional image rendering. We realize differentiable optimization of the omnidirectional radiance field without the requirement of cube-map rectification or tangent-plane approximation. Extensive experiments conducted in egocentric and roaming scenarios demonstrate that our method achieves state-of-the-art reconstruction quality and high rendering speed using omnidirectional images.
Introduction
Photorealistic 3D reconstruction has been revolutionized by Neural Radiance Fields (NeRF) and, more recently, 3D Gaussian Splatting (3DGS). However, most existing systems are optimized for standard perspective cameras. When applied to omnidirectional (360-degree) imagery, researchers typically face a trade-off: NeRF-based methods handle 360-degree views well but suffer from slow training and rendering speeds, while perspective-based 3DGS requires computationally expensive "rectification" into cube-maps or tangent-plane approximations. These approximations often introduce distortions and fail to capture the full environmental context in a single optimization step.
OmniGS addresses these challenges by bringing the lightning-fast performance of 3DGS to 360-degree imagery. By natively supporting omnidirectional geometry, OmniGS achieves state-of-the-art reconstruction quality and real-time rendering speeds (exceeding 100 FPS) for large-scale indoor and egocentric scenes.
Methods
The primary innovation of OmniGS is a GPU-accelerated omnidirectional rasterizer that splats Gaussians directly onto equirectangular screen space. This approach avoids the overhead of warping images to fit perspective models by adapting the underlying mathematics to the spherical camera model.
Spherical Derivatives and Direct Splatting
The authors derived the Jacobian matrix required to project 3D Gaussians directly onto an equirectangular (latitude-longitude) image plane. Unlike previous methods that approximate 360-degree views using multiple perspective tiles, OmniGS splats Gaussians directly into the equirectangular space, maintaining high fidelity and allowing the model to learn from the entire environment simultaneously.
Radial Sorting
In standard 3DGS, Gaussians are sorted by their depth ($z$-coordinate) for alpha-blending. In a spherical coordinate system, "depth" is relative to the center. OmniGS implements radial sorting based on the distance ($t_r$) from the camera center, which is physically more accurate for omnidirectional viewing. The entire pipeline is end-to-end differentiable, enabling the optimization of position, rotation, scale, and opacity for millions of Gaussians.
Results and Discussion
OmniGS was evaluated on two primary benchmarks: 360Roam (complex real-world indoor scenes) and EgoNeRF (including synthetic OmniBlender and real-world Ricoh360 datasets).
Performance Metrics
OmniGS significantly outperforms previous NeRF-based 360 reconstruction methods in both quality and speed:
- Rendering Speed: On the 360Roam dataset, OmniGS achieves 121 FPS, a massive improvement over the 30 FPS of 360Roam-NeRF and the <1 FPS of standard NeRF.
- Training Efficiency: OmniGS reaches state-of-the-art quality in less than 25 minutes (32k iterations), whereas NeRF-based baselines often require several hours.
- Visual Quality: On the OmniBlender indoor dataset, OmniGS improved the PSNR by over 5.1 dB compared to EgoNeRF while running 57 times faster.
Global Context Advantage
By processing the entire 360-degree field of view in every training iteration, OmniGS reaches a stable, high-quality reconstruction faster than perspective-based methods. Even when compared against standard 3DGS (using cropped perspective images), OmniGS performs better because its training process utilizes the full environmental context, leading to more robust Gaussian placement.
Conclusion
OmniGS demonstrates that the local affine approximation of Gaussian Splatting can be successfully adapted to non-linear camera models like equirectangular projection. While equirectangular images naturally exhibit distortion at the poles, OmniGS handles these regions more effectively than perspective-based methods. Future work includes extending the system to dynamic scenes and integrating it into omnidirectional visual SLAM systems for real-time, photorealistic robotic mapping.
Summary updated