ODGS: 3D Scene Reconstruction from Omnidirectional Images with 3D Gaussian Splattings

Suyoung Lee, Jaeyoung Chung, Jaeyoo Huh, Kyoung Mu Lee

Abstract

Omnidirectional (or 360-degree) images are increasingly being used for 3D applications since they allow the rendering of an entire scene with a single image. Existing works based on neural radiance fields demonstrate successful 3D reconstruction quality on egocentric videos, yet they suffer from long training and rendering times. Recently, 3D Gaussian splatting has gained attention for its fast optimization and real-time rendering. However, directly using a perspective rasterizer to omnidirectional images results in severe distortion due to the different optical properties between two image domains. In this work, we present ODGS, a novel rasterization pipeline for omnidirectional images, with geometric interpretation. For each Gaussian, we define a tangent plane that touches the unit sphere and is perpendicular to the ray headed toward the Gaussian center. We then leverage a perspective camera rasterizer to project the Gaussian onto the corresponding tangent plane. The projected Gaussians are transformed and combined into the omnidirectional image, finalizing the omnidirectional rasterization process. The entire rasterization process is parallelized using CUDA, achieving optimization and rendering speeds 100 times faster than NeRF-based methods.

Introduction

Omnidirectional (360-degree) images provide a comprehensive view of an environment in a single capture, making them ideal for 3D scene reconstruction. However, standard 3D reconstruction techniques often struggle with the unique geometry of these images. While Neural Radiance Fields (NeRFs) can produce high-quality results, they are computationally expensive and slow to train. 3D Gaussian Splatting (3DGS) offers a faster alternative but is designed for perspective cameras. Applying 3DGS directly to 360-degree images—typically represented in equirectangular projection (ERP)—leads to significant distortions because the linear projection math of perspective cameras does not align with spherical geometry.

The ODGS Pipeline

ODGS addresses these challenges by introducing a novel omnidirectional rasterization pipeline. Instead of a single global projection, the method employs a local geometric approach to minimize distortion.

The ODGS rasterization pipeline: (a) Coordinate transformation, (b) Tangent plane projection, (c) Horizontal stretching in equirectangular space, and (d) Final rescaling and combination.

The process involves four key stages:

  1. Coordinate Transformation: Aligning the Gaussian centers within the spherical coordinate system.
  2. Tangent Plane Projection: For each 3D Gaussian, a unique tangent plane is defined at its location on a unit sphere. The Gaussian is then projected onto this plane using a standard perspective model, which is locally accurate and minimizes distortion.
  3. Equirectangular Mapping: The projected Gaussians are transformed into the equirectangular space. This includes horizontal stretching to account for the varying pixel density of ERP images.
  4. Rescaling and Combination: The final image is synthesized by combining these local projections.

To handle the extreme stretching at the poles (zenith and nadir), ODGS implements an elevation-aware dynamic densification rule. This ensures that computational resources are distributed efficiently across the spherical domain.

Experimental Results

The authors evaluated ODGS across six datasets, including egocentric (e.g., OmniBlender, Ricoh360) and roaming scenarios (e.g., 360Roam, OmniScenes).

Comparison of PSNR, SSIM, and LPIPS over training time for the OmniBlender and OmniPhotos datasets. ODGS (purple line) achieves higher quality significantly faster than competing methods.

ODGS consistently outperforms NeRF-based methods like EgoNeRF and TensoRF in both reconstruction quality (PSNR, SSIM) and perceptual metrics (LPIPS). Crucially, ODGS achieves a massive speedup, rendering frames in approximately 0.03 seconds—roughly 100 times faster than NeRF-based alternatives. This performance enables real-time interaction with reconstructed 360-degree scenes.

Conclusion

ODGS successfully adapts 3D Gaussian Splatting for omnidirectional imagery by leveraging local tangent plane projections. This approach is mathematically rigorous and computationally efficient, providing a significant leap in both speed and visual fidelity. While the current implementation uses a local affine approximation, future work could explore non-linear distributions to further refine accuracy in regions of high curvature.

Summary updated

Abstract

Omnidirectional (or 360-degree) images are increasingly being used for 3D applications since they allow the rendering of an entire scene with a single image. Existing works based on neural radiance fields demonstrate successful 3D reconstruction quality on egocentric videos, yet they suffer from long training and rendering times. Recently, 3D Gaussian splatting has gained attention for its fast optimization and real-time rendering. However, directly using a perspective rasterizer to omnidirectional images results in severe distortion due to the different optical properties between two image domains. In this work, we present ODGS, a novel rasterization pipeline for omnidirectional images, with geometric interpretation. For each Gaussian, we define a tangent plane that touches the unit sphere and is perpendicular to the ray headed toward the Gaussian center. We then leverage a perspective camera rasterizer to project the Gaussian onto the corresponding tangent plane. The projected Gaussians are transformed and combined into the omnidirectional image, finalizing the omnidirectional rasterization process. This interpretation reveals the implicit assumptions within the proposed pipeline, which we verify through mathematical proofs. The entire rasterization process is parallelized using CUDA, achieving optimization and rendering speeds 100 times faster than NeRF-based methods. Our comprehensive experiments highlight the superiority of ODGS by delivering the best reconstruction and perceptual quality across various datasets. Additionally, results on roaming datasets demonstrate that ODGS restores fine details effectively, even when reconstructing large 3D scenes. The source code is available on our project page (https://github.com/esw0116/ODGS).
BibTeX
@inproceedings{lee2024odgs,
  author = {Suyoung Lee and Jaeyoung Chung and Jaeyoo Huh and Kyoung Mu Lee},
  title = {{ODGS: 3D Scene Reconstruction from Omnidirectional Images with 3D Gaussian Splattings}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=CovjSQmNOD},
}
Figure