TL;DR
OmniSplat is a training-free, feed-forward framework that enables high-quality 3D Gaussian Splatting (3DGS) for 360° omnidirectional images. By decomposing distorted spherical images into a "quasi-uniform" Yin-Yang grid, the authors successfully repurpose powerful models trained on standard perspective images to reconstruct entire 3D scenes in under a second.
What Problem Does This Paper Address?
While 3D Gaussian Splatting has revolutionized real-time 3D reconstruction, most "feed-forward" models—those that generate a scene instantly without hours of per-scene optimization—are designed for standard perspective cameras.
Omnidirectional (360°) images are ideal for capturing entire environments, but they pose two major hurdles for existing AI models:
- Geometric Distortion: Standard equirectangular projections (the "flat map" view of a sphere) stretch objects severely at the poles. Standard neural network encoders, trained on normal photos, fail to understand these distorted shapes.
- Sampling Artifacts: When rendering novel views from 360° data, the non-uniform density of points near the poles often creates "stripe-like" artifacts or dark spots.
Because there isn't enough multi-view 360° data to train a massive transformer model from scratch, we need a way to "tame" existing perspective-based models to work in a spherical world.
Methods
The core innovation of OmniSplat is the use of the Yin-Yang grid to bridge the domain gap.
- Yin-Yang Decomposition: Instead of using a single distorted rectangular map, the authors split the sphere into two overlapping square grids (Yin and Yang), similar to the seams on a tennis ball. These grids are "quasi-uniform," meaning they look much more like standard perspective photos.
- Feature Extraction & Attention: The system feeds these Yin-Yang images into a pre-trained perspective model (specifically MVSplat). By performing cross-view attention across these grids, the model can estimate depth and Gaussian parameters (position, opacity, color) for the whole scene.
- Yin-Yang Rasterization: To render a new view, the model doesn't render a 360° image directly. Instead, it renders a Yin image and a Yang image separately and then blends them in pixel space. This prevents the "pole artifacts" common in spherical rendering.
- Training-Free Transfer: Remarkably, OmniSplat uses the weights of a model trained on perspective images without any additional fine-tuning, proving that the Yin-Yang decomposition effectively "tricks" the model into thinking it's looking at standard photos.
Datasets
The authors evaluated OmniSplat across six diverse benchmarks:
- Synthetic: OmniBlender (11 synthetic indoor scenes).
- Real-World (Small Motion): Ricoh360 and OmniPhotos (outdoor scenes with 360° cameras).
- Real-World (Large Motion): 360Roam, OmniScenes, and 360VO.
They established new evaluation protocols for these datasets to test "sparse-view" reconstruction, where the model only sees two 360° images and must predict the rest of the scene.
Key Insights and Ideas
- The "Quasi-Uniform" Advantage: Unlike cubemaps (which have sharp edges) or icosahedral grids (which use triangles), the Yin-Yang grid uses squares. This makes it natively compatible with standard Convolutional Neural Networks (CNNs) while maintaining a nearly uniform distribution of pixels across the sphere.
- Instant Initialization: The authors found that their feed-forward prediction provides such a good "starting point" that just 100 iterations of optimization (taking ~11 seconds) can outperform traditional methods that optimize for 30 minutes.
- Attention as a Segmenter: By looking at the attention scores generated during the reconstruction process, the model can identify consistent objects across different views. This allows for "pixel-aligned" 3D editing, such as cleanly removing an object from a 3D scene.
Key Results
OmniSplat achieves a state-of-the-art balance between speed and image quality.
- Speed: It is the fastest model tested, with a runtime of just 0.532 seconds for a full reconstruction.
- Accuracy: On the OmniBlender dataset, it achieved a PSNR of 21.22 dB, significantly higher than other feed-forward baselines like MVSplat (16.97 dB) or PixelSplat (16.97 dB) when applied to 360° data.
- OmniSplat+opt: With a tiny bit of optimization (12 seconds total), the model reaches 22.47 dB, surpassing optimization-only methods (ODGS) that take 1,920 seconds (32 minutes).
Qualitatively, OmniSplat avoids the blurriness and "ghosting" seen in perspective-based models and the "stripe" artifacts seen in previous omnidirectional attempts.
Limitations and Future Work
The authors note a few areas for improvement:
- Interpolation Loss: Converting the Yang grid back to a standard equirectangular format requires horizontal interpolation, which can slightly degrade image quality.
- Adaptation Speed: While 11 seconds is fast, the authors believe "meta-learning" could reduce the need for any per-scene optimization steps even further.
- Complex Geometry: Like many 3DGS methods, very thin structures or highly reflective surfaces remain a challenge for sparse-view reconstruction.
Summary updated