OmniGS: Omnidirectional Gaussian Splatting for Fast Radiance Field Reconstruction using Omnidirectional Images

Longwei Li, Huajian Huang, Sai-Kit Yeung, Hui Cheng

TL;DR

This paper introduces OmniGS, a novel method that adapts 3D Gaussian Splatting for omnidirectional images, enabling faster and more detailed 3D scene reconstruction. By developing a specialized omnidirectional rasterizer, OmniGS achieves state-of-the-art photorealistic results and high rendering speeds, overcoming limitations of previous perspective-only approaches.

What Problem Does This Paper Address?

Reconstructing realistic 3D scenes from images is crucial for applications like virtual reality and robotics. While recent methods like 3D Gaussian Splatting (3DGS) have dramatically improved reconstruction speed and quality, they are primarily designed for standard "perspective" images. This means they can't directly leverage the rich, all-encompassing information captured by omnidirectional (360-degree) cameras. Using omnidirectional images with existing 3DGS would require computationally expensive preprocessing steps like cube-map rectification or tangent-plane approximations, negating the speed benefits. OmniGS aims to bridge this gap by enabling direct, efficient, and high-quality radiance field reconstruction using omnidirectional images.

Methods

The core innovation of OmniGS lies in its omnidirectional rasterizer. Unlike traditional 3DGS that projects Gaussians onto a perspective view, OmniGS directly splats 3D Gaussians onto an equirectangular projection screen space. This is achieved through:

  • Theoretical Analysis: The authors derived the necessary mathematical derivatives for spherical camera models within the 3D Gaussian Splatting framework. This is crucial for enabling differentiable rendering with omnidirectional inputs.
  • GPU-Accelerated Rasterizer: A new, efficient rasterizer was developed that directly handles the equirectangular projection. This avoids the need for intermediate representations like cube maps, significantly speeding up the process.
  • Differentiable Optimization: The system allows for end-to-end differentiable optimization of the radiance field without relying on approximations or rectifications, directly learning from omnidirectional images.
  • Standard 3DGS Components: OmniGS builds upon the foundational principles of 3D Gaussian Splatting, including representing scenes with 3D Gaussians (position, color, scale, rotation, opacity) and using an alpha-blending model for rendering.

Datasets

The paper evaluates OmniGS on two primary datasets:

  • 360Roam dataset: This dataset features 10 indoor scenes captured by a mobile robot using a commercial omnidirectional camera.
  • EgoNeRF dataset: This dataset includes both simulated (OmniBlender) and real-world (Ricoh360) egocentric scenes.

The authors note that for the 360Roam dataset, the bottom part of the images (corresponding to the robot's base) was excluded from loss computation due to the dynamic nature of the robot. For EgoNeRF, they used provided classifications and original image resolutions.

Key Insights and Ideas

  • Direct Equirectangular Splatting is Key: The most significant insight is that by directly deriving and implementing the necessary mathematical transformations for equirectangular projection, one can achieve efficient and high-quality omnidirectional radiance field reconstruction without the computational overhead of traditional methods.
  • Omnidirectional Cameras Unlock Speed: Leveraging the full field-of-view of omnidirectional images from the start, rather than processing them as multiple perspective views, leads to substantial speedups in both training and rendering.
  • Unified Representation for Better Results: When comparing OmniGS with perspective 3DGS, it's shown that unifying the testing image format (either all omnidirectional or all perspective) leads to more meaningful comparisons and highlights OmniGS's strengths.
  • Scalability Beyond Single Rooms: Unlike some prior methods that rely on indoor layout priors, OmniGS's direct screen-space splatting approach makes it applicable to diverse indoor and outdoor scenes without specific scene assumptions.

Key Results

  • State-of-the-Art Quality: OmniGS achieves state-of-the-art (SOTA) reconstruction quality on both the 360Roam and EgoNeRF datasets, outperforming existing NeRF-based and 3DGS methods in terms of PSNR, SSIM, and LPIPS metrics.
  • Significant Speedup: OmniGS offers substantially faster rendering speeds (FPS) compared to baseline methods. For instance, it achieves over 4x higher FPS than 360Roam baselines and up to 58x higher FPS on EgoNeRF scenes.
  • Faster Training: The method also demonstrates faster training times. For example, it achieved comparable quality to 360Roam in under 25 minutes, while baselines took hours.
  • Improved Perspective Views: When rendered omnidirectional views are cropped into perspective images, OmniGS produces sharper details and fewer artifacts compared to perspective 3DGS, especially when trained for the same duration.

Limitations and Future Work

  • Limitations:
    • The paper notes that when comparing "tested-as-trained" performance, perspective 3DGS was slightly better than OmniGS when both were trained and tested on perspective images. This is attributed to potential near-pole distortion in real omnidirectional cameras affecting quantitative results.
  • The authors mention skipping the bottom part of images in 360Roam scenes due to the dynamic base of the mobile robot, implying that dynamic scene handling is not addressed.
  • Future Work:
    • The authors suggest that OmniGS holds potential for integration into omnidirectional visual SLAM systems to enable real-time online photorealistic mapping.
  • The method's ability to generate better perspective views from omnidirectional reconstructions suggests its strong scalability for various applications.

Summary updated

Abstract

Photorealistic reconstruction relying on 3D Gaussian Splatting has shown promising potential in various domains. However, the current 3D Gaussian Splatting system only supports radiance field reconstruction using undistorted perspective images. In this paper, we present OmniGS, a novel omnidirectional Gaussian splatting system, to take advantage of omnidirectional images for fast radiance field reconstruction. Specifically, we conduct a theoretical analysis of spherical camera model derivatives in 3D Gaussian Splatting. According to the derivatives, we then implement a new GPU-accelerated omnidirectional rasterizer that directly splats 3D Gaussians onto the equirectangular screen space for omnidirectional image rendering. We realize differentiable optimization of the omnidirectional radiance field without the requirement of cube-map rectification or tangent-plane approximation. Extensive experiments conducted in egocentric and roaming scenarios demonstrate that our method achieves state-of-the-art reconstruction quality and high rendering speed using omnidirectional images. The code will be publicly available.
BibTeX
@inproceedings{li2025omnigs,
  author = {Longwei Li and Huajian Huang and Sai-Kit Yeung and Hui Cheng},
  title = {{OmniGS: Omnidirectional Gaussian Splatting for Fast Radiance Field Reconstruction using Omnidirectional Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=hoQOqSjRWC},
}
Figure