Abstract
Omnidirectional (or 360-degree) images are increasingly being used for 3D applications since they allow the rendering of an entire scene with a single image. Existing works based on neural radiance fields demonstrate successful 3D reconstruction quality on egocentric videos, yet they suffer from long training and rendering times. Recently, 3D Gaussian splatting has gained attention for its fast optimization and real-time rendering. However, directly using a perspective rasterizer to omnidirectional images results in severe distortion due to the different optical properties between two image domains. In this work, we present ODGS, a novel rasterization pipeline for omnidirectional images, with geometric interpretation. For each Gaussian, we define a tangent plane that touches the unit sphere and is perpendicular to the ray headed toward the Gaussian center. We then leverage a perspective camera rasterizer to project the Gaussian onto the corresponding tangent plane. The projected Gaussians are transformed and combined into the omnidirectional image, finalizing the omnidirectional rasterization process. The entire rasterization process is parallelized using CUDA, achieving optimization and rendering speeds 100 times faster than NeRF-based methods.
Introduction
Omnidirectional (360-degree) images provide a comprehensive view of an environment in a single capture, making them ideal for 3D scene reconstruction. However, standard 3D reconstruction techniques often struggle with the unique geometry of these images. While Neural Radiance Fields (NeRFs) can produce high-quality results, they are computationally expensive and slow to train. 3D Gaussian Splatting (3DGS) offers a faster alternative but is designed for perspective cameras. Applying 3DGS directly to 360-degree images—typically represented in equirectangular projection (ERP)—leads to significant distortions because the linear projection math of perspective cameras does not align with spherical geometry.
The ODGS Pipeline
ODGS addresses these challenges by introducing a novel omnidirectional rasterization pipeline. Instead of a single global projection, the method employs a local geometric approach to minimize distortion.
The process involves four key stages:
- Coordinate Transformation: Aligning the Gaussian centers within the spherical coordinate system.
- Tangent Plane Projection: For each 3D Gaussian, a unique tangent plane is defined at its location on a unit sphere. The Gaussian is then projected onto this plane using a standard perspective model, which is locally accurate and minimizes distortion.
- Equirectangular Mapping: The projected Gaussians are transformed into the equirectangular space. This includes horizontal stretching to account for the varying pixel density of ERP images.
- Rescaling and Combination: The final image is synthesized by combining these local projections.
To handle the extreme stretching at the poles (zenith and nadir), ODGS implements an elevation-aware dynamic densification rule. This ensures that computational resources are distributed efficiently across the spherical domain.
Experimental Results
The authors evaluated ODGS across six datasets, including egocentric (e.g., OmniBlender, Ricoh360) and roaming scenarios (e.g., 360Roam, OmniScenes).
ODGS consistently outperforms NeRF-based methods like EgoNeRF and TensoRF in both reconstruction quality (PSNR, SSIM) and perceptual metrics (LPIPS). Crucially, ODGS achieves a massive speedup, rendering frames in approximately 0.03 seconds—roughly 100 times faster than NeRF-based alternatives. This performance enables real-time interaction with reconstructed 360-degree scenes.
Conclusion
ODGS successfully adapts 3D Gaussian Splatting for omnidirectional imagery by leveraging local tangent plane projections. This approach is mathematically rigorous and computationally efficient, providing a significant leap in both speed and visual fidelity. While the current implementation uses a local affine approximation, future work could explore non-linear distributions to further refine accuracy in regions of high curvature.
Summary updated