TL;DR
This paper introduces OmniGS, a novel method that adapts 3D Gaussian Splatting for omnidirectional images, enabling faster and more detailed 3D scene reconstruction. By developing a specialized omnidirectional rasterizer, OmniGS achieves state-of-the-art photorealistic results and high rendering speeds, overcoming limitations of previous perspective-only approaches.
What Problem Does This Paper Address?
Reconstructing realistic 3D scenes from images is crucial for applications like virtual reality and robotics. While recent methods like 3D Gaussian Splatting (3DGS) have dramatically improved reconstruction speed and quality, they are primarily designed for standard "perspective" images. This means they can't directly leverage the rich, all-encompassing information captured by omnidirectional (360-degree) cameras. Using omnidirectional images with existing 3DGS would require computationally expensive preprocessing steps like cube-map rectification or tangent-plane approximations, negating the speed benefits. OmniGS aims to bridge this gap by enabling direct, efficient, and high-quality radiance field reconstruction using omnidirectional images.
Methods
The core innovation of OmniGS lies in its omnidirectional rasterizer. Unlike traditional 3DGS that projects Gaussians onto a perspective view, OmniGS directly splats 3D Gaussians onto an equirectangular projection screen space. This is achieved through:
- Theoretical Analysis: The authors derived the necessary mathematical derivatives for spherical camera models within the 3D Gaussian Splatting framework. This is crucial for enabling differentiable rendering with omnidirectional inputs.
- GPU-Accelerated Rasterizer: A new, efficient rasterizer was developed that directly handles the equirectangular projection. This avoids the need for intermediate representations like cube maps, significantly speeding up the process.
- Differentiable Optimization: The system allows for end-to-end differentiable optimization of the radiance field without relying on approximations or rectifications, directly learning from omnidirectional images.
- Standard 3DGS Components: OmniGS builds upon the foundational principles of 3D Gaussian Splatting, including representing scenes with 3D Gaussians (position, color, scale, rotation, opacity) and using an alpha-blending model for rendering.
Datasets
The paper evaluates OmniGS on two primary datasets:
- 360Roam dataset: This dataset features 10 indoor scenes captured by a mobile robot using a commercial omnidirectional camera.
- EgoNeRF dataset: This dataset includes both simulated (OmniBlender) and real-world (Ricoh360) egocentric scenes.
The authors note that for the 360Roam dataset, the bottom part of the images (corresponding to the robot's base) was excluded from loss computation due to the dynamic nature of the robot. For EgoNeRF, they used provided classifications and original image resolutions.
Key Insights and Ideas
- Direct Equirectangular Splatting is Key: The most significant insight is that by directly deriving and implementing the necessary mathematical transformations for equirectangular projection, one can achieve efficient and high-quality omnidirectional radiance field reconstruction without the computational overhead of traditional methods.
- Omnidirectional Cameras Unlock Speed: Leveraging the full field-of-view of omnidirectional images from the start, rather than processing them as multiple perspective views, leads to substantial speedups in both training and rendering.
- Unified Representation for Better Results: When comparing OmniGS with perspective 3DGS, it's shown that unifying the testing image format (either all omnidirectional or all perspective) leads to more meaningful comparisons and highlights OmniGS's strengths.
- Scalability Beyond Single Rooms: Unlike some prior methods that rely on indoor layout priors, OmniGS's direct screen-space splatting approach makes it applicable to diverse indoor and outdoor scenes without specific scene assumptions.
Key Results
- State-of-the-Art Quality: OmniGS achieves state-of-the-art (SOTA) reconstruction quality on both the 360Roam and EgoNeRF datasets, outperforming existing NeRF-based and 3DGS methods in terms of PSNR, SSIM, and LPIPS metrics.
- Significant Speedup: OmniGS offers substantially faster rendering speeds (FPS) compared to baseline methods. For instance, it achieves over 4x higher FPS than 360Roam baselines and up to 58x higher FPS on EgoNeRF scenes.
- Faster Training: The method also demonstrates faster training times. For example, it achieved comparable quality to 360Roam in under 25 minutes, while baselines took hours.
- Improved Perspective Views: When rendered omnidirectional views are cropped into perspective images, OmniGS produces sharper details and fewer artifacts compared to perspective 3DGS, especially when trained for the same duration.
Limitations and Future Work
- Limitations:
- The paper notes that when comparing "tested-as-trained" performance, perspective 3DGS was slightly better than OmniGS when both were trained and tested on perspective images. This is attributed to potential near-pole distortion in real omnidirectional cameras affecting quantitative results.
- The authors mention skipping the bottom part of images in 360Roam scenes due to the dynamic base of the mobile robot, implying that dynamic scene handling is not addressed.
- Future Work:
- The authors suggest that OmniGS holds potential for integration into omnidirectional visual SLAM systems to enable real-time online photorealistic mapping.
- The method's ability to generate better perspective views from omnidirectional reconstructions suggests its strong scalability for various applications.
Summary updated