OmniSplat: Taming Feed-Forward 3D Gaussian Splatting for Omnidirectional Images with Editable Capabilities

Suyoung Lee, Jaeyoung Chung, Kihoon Kim, Jaeyoo Huh, Gunhee Lee, Minsoo Lee, Kyoung Mu Lee

TL;DR

OmniSplat is a training-free, feed-forward framework that enables high-quality 3D Gaussian Splatting (3DGS) for 360° omnidirectional images. By decomposing distorted spherical images into a "quasi-uniform" Yin-Yang grid, the authors successfully repurpose powerful models trained on standard perspective images to reconstruct entire 3D scenes in under a second.

What Problem Does This Paper Address?

While 3D Gaussian Splatting has revolutionized real-time 3D reconstruction, most "feed-forward" models—those that generate a scene instantly without hours of per-scene optimization—are designed for standard perspective cameras.

Omnidirectional (360°) images are ideal for capturing entire environments, but they pose two major hurdles for existing AI models:

  1. Geometric Distortion: Standard equirectangular projections (the "flat map" view of a sphere) stretch objects severely at the poles. Standard neural network encoders, trained on normal photos, fail to understand these distorted shapes.
  2. Sampling Artifacts: When rendering novel views from 360° data, the non-uniform density of points near the poles often creates "stripe-like" artifacts or dark spots.

Because there isn't enough multi-view 360° data to train a massive transformer model from scratch, we need a way to "tame" existing perspective-based models to work in a spherical world.

Methods

The core innovation of OmniSplat is the use of the Yin-Yang grid to bridge the domain gap.

  • Yin-Yang Decomposition: Instead of using a single distorted rectangular map, the authors split the sphere into two overlapping square grids (Yin and Yang), similar to the seams on a tennis ball. These grids are "quasi-uniform," meaning they look much more like standard perspective photos.
  • Feature Extraction & Attention: The system feeds these Yin-Yang images into a pre-trained perspective model (specifically MVSplat). By performing cross-view attention across these grids, the model can estimate depth and Gaussian parameters (position, opacity, color) for the whole scene.
  • Yin-Yang Rasterization: To render a new view, the model doesn't render a 360° image directly. Instead, it renders a Yin image and a Yang image separately and then blends them in pixel space. This prevents the "pole artifacts" common in spherical rendering.
  • Training-Free Transfer: Remarkably, OmniSplat uses the weights of a model trained on perspective images without any additional fine-tuning, proving that the Yin-Yang decomposition effectively "tricks" the model into thinking it's looking at standard photos.
Figure 2. The overall process of OmniSplat. The two reference omnidirectional images are decomposed into Yin-Yang images, and the cross-view attention is conducted across grids along with epipolar lines to compose cost volume. The 3DGS parameters are estimated and Yin-Yang images are rasterized from the novel view. The two images are combined to synthesize the final omnidirectional image. In cross-view attention, we present red and yellow points and the corresponding sphere sweep curves with the same color. Each image performs cross-attention to the Yin-Yang images from other views, following geometric constraints.

Datasets

The authors evaluated OmniSplat across six diverse benchmarks:

  • Synthetic: OmniBlender (11 synthetic indoor scenes).
  • Real-World (Small Motion): Ricoh360 and OmniPhotos (outdoor scenes with 360° cameras).
  • Real-World (Large Motion): 360Roam, OmniScenes, and 360VO.

They established new evaluation protocols for these datasets to test "sparse-view" reconstruction, where the model only sees two 360° images and must predict the rest of the scene.

Key Insights and Ideas

  • The "Quasi-Uniform" Advantage: Unlike cubemaps (which have sharp edges) or icosahedral grids (which use triangles), the Yin-Yang grid uses squares. This makes it natively compatible with standard Convolutional Neural Networks (CNNs) while maintaining a nearly uniform distribution of pixels across the sphere.
  • Instant Initialization: The authors found that their feed-forward prediction provides such a good "starting point" that just 100 iterations of optimization (taking ~11 seconds) can outperform traditional methods that optimize for 30 minutes.
  • Attention as a Segmenter: By looking at the attention scores generated during the reconstruction process, the model can identify consistent objects across different views. This allows for "pixel-aligned" 3D editing, such as cleanly removing an object from a 3D scene.

Key Results

OmniSplat achieves a state-of-the-art balance between speed and image quality.

  • Speed: It is the fastest model tested, with a runtime of just 0.532 seconds for a full reconstruction.
  • Accuracy: On the OmniBlender dataset, it achieved a PSNR of 21.22 dB, significantly higher than other feed-forward baselines like MVSplat (16.97 dB) or PixelSplat (16.97 dB) when applied to 360° data.
  • OmniSplat+opt: With a tiny bit of optimization (12 seconds total), the model reaches 22.47 dB, surpassing optimization-only methods (ODGS) that take 1,920 seconds (32 minutes).
Figure 1. PSNR-runtime trade-off. A chart of reconstruction PSNR-runtime trade-off in novel view image on OmniBlender [7]. OmniSplat shows the best trade-off compared to the original feed- forward networks for perspective images.

Qualitatively, OmniSplat avoids the blurriness and "ghosting" seen in perspective-based models and the "stripe" artifacts seen in previous omnidirectional attempts.

Figure 3. Qualitative comparison. Novel view synthesized image examples in various datasets. Each scene is brought from OmniBlender, Ricoh360, and OmniPhotos, respectively. Best viewed when zoomed in.

Limitations and Future Work

The authors note a few areas for improvement:

  • Interpolation Loss: Converting the Yang grid back to a standard equirectangular format requires horizontal interpolation, which can slightly degrade image quality.
  • Adaptation Speed: While 11 seconds is fast, the authors believe "meta-learning" could reduce the need for any per-scene optimization steps even further.
  • Complex Geometry: Like many 3DGS methods, very thin structures or highly reflective surfaces remain a challenge for sparse-view reconstruction.

Summary updated

Abstract

Feed-forward 3D Gaussian splatting (3DGS) models have gained significant popularity due to their ability to generate scenes immediately without needing per-scene optimization. Although omnidirectional images are becoming more popular since they reduce the computation required for image stitching to composite a holistic scene, existing feed-forward models are only designed for perspective images. The unique optical properties of omnidirectional images make it difficult for feature encoders to correctly understand the context of the image and make the Gaussian non-uniform in space, which hinders the image quality synthesized from novel views. We propose OmniSplat, a training-free fast feed-forward 3DGS generation framework for omnidirectional images. We adopt a Yin-Yang grid and decompose images based on it to reduce the domain gap between omnidirectional and perspective images. The Yin-Yang grid can use the existing CNN structure as it is, but its quasi-uniform characteristic allows the decomposed image to be similar to a perspective image, so it can exploit the strong prior knowledge of the learned feed-forward network. OmniSplat demonstrates higher reconstruction accuracy than existing feed-forward networks trained on perspective images. Our project page is available on: https://robot0321.github.io/omnisplat/index.html.
BibTeX
@inproceedings{lee2025omnisplat,
  author = {Suyoung Lee and Jaeyoung Chung and Kihoon Kim and Jaeyoo Huh and Gunhee Lee and Minsoo Lee and Kyoung Mu Lee},
  title = {{OmniSplat: Taming Feed-Forward 3D Gaussian Splatting for Omnidirectional Images with Editable Capabilities}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=u8aC6LBquX},
}
Figure