Introduction
Capturing large-scale environments for Virtual Reality (VR), Augmented Reality (AR), or robotics typically requires extensive photographic coverage. 360-degree cameras offer a significant efficiency advantage by capturing the entire surrounding environment in a single shot. However, these cameras output images using Equirectangular Projection (ERP), which introduces severe distortion, particularly near the poles.
Standard 3D Gaussian Splatting (3DGS) algorithms struggle with ERP images, often generating oversized, inaccurate Gaussians that degrade rendering quality. Additionally, 360-degree captures frequently include the camera rig or photographer, which move relative to the scene and disrupt geometric consistency. ErpGS (Omnidirectional Gaussian Splatting) addresses these challenges by adapting 3DGS specifically for the ERP camera model through geometric and scale regularizations, distortion-aware weighting, and dynamic masking.
Methodology: Adapting 3DGS for Omnidirectional Vision
ErpGS introduces several key techniques to mitigate the artifacts inherent in equirectangular 3D reconstruction.
Geometric and Scale Regularization
To ensure coherent 3D geometry and prevent "floating" artifacts, ErpGS enforces a consistency between normal vectors rendered from 3D Gaussians and those calculated from the rendered depth map. Unlike standard 2D grids, ErpGS calculates depth-normals on a spherical tangent plane to account for ERP distortion. Furthermore, a scale regularization term penalizes excessively large Gaussians and encourages them to remain flat, preventing the model from using "blobs" to compensate for image stretching.
Distortion-Aware Optimization
A critical insight of ErpGS is that treating an ERP image like a standard perspective photo is insufficient. The stretching at high latitudes means that pixels near the poles represent smaller angular areas than those at the equator. ErpGS introduces distortion-aware weights that prioritize the less-distorted equatorial regions during training, which ablation studies have shown to be the most vital component for maintaining high rendering accuracy.
Viewpoint-Dependent Masking
To handle dynamic obstacles like tripod legs or the photographer, ErpGS employs viewpoint-dependent masks. By ignoring these transient elements during optimization, the model avoids creating conflicting geometry that would otherwise manifest as messy "floaters" in the reconstructed scene.
Experimental Evaluation
The authors evaluated ErpGS across three diverse datasets:
- OmniBlender: Synthetic indoor and outdoor scenes.
- Ricoh 360: Real-world outdoor scenes captured via an egocentric spiral path.
- OmniScenes: Real-world indoor scenes from varied stationary positions.
Quantitative and Qualitative Results
ErpGS consistently outperformed state-of-the-art methods such as EgoNeRF, ODGS, and OmniGS. In the "barbershop" scene from OmniBlender, ErpGS achieved a PSNR of 38.71 dB, significantly higher than OmniGS (37.26 dB) and EgoNeRF (30.57 dB).
As shown in the visual comparisons, ErpGS successfully eliminates the cloudy artifacts and geometric inconsistencies common in other methods, particularly in regions previously occupied by camera hardware. Ablation tests confirmed the necessity of each component: removing distortion weights caused the most significant performance drop (from 35.16 dB to 33.86 dB on OmniBlender), while removing regularizations also led to measurable fidelity losses.
Conclusion and Future Directions
ErpGS represents a significant step forward in photorealistic 3D reconstruction from panoramic imagery. By explicitly modeling the spherical nature of ERP data, it overcomes the limitations of standard 3DGS.
However, the method remains dependent on the quality of initial Structure-from-Motion (SfM) point clouds. In expansive outdoor scenes, such as the 'lone-monk' dataset, ErpGS may fail to render distant objects if the initial sparse point cloud is insufficient. Future research will likely focus on improving geometry initialization and optimization for distant background elements where sparse data is lacking.
Summary updated