OmniSplat: Taming Feed-Forward 3D Gaussian Splatting for Omnidirectional Images with Editable Capabilities

Suyoung Lee, Jaeyoung Chung, Kihoon Kim, Jaeyoo Huh, Gunhee Lee, Minsoo Lee, Kyoung Mu Lee

Abstract

Feed-forward 3D Gaussian Splatting (3DGS) models have gained significant popularity due to their ability to generate scenes immediately without needing per-scene optimization. Although omnidirectional images are getting more popular since they reduce the computation for image stitching to composite a holistic scene, existing feed-forward models are only designed for perspective images. The unique optical properties of omnidirectional images make it difficult for feature encoders to correctly understand the context of the image and make the Gaussian non-uniform in space, which hinders the image quality synthesized from novel views. We propose OmniSplat, a pioneering work for fast feed-forward 3DGS generation from a few omnidirectional images. We introduce Yin-Yang grid and decompose images based on it to reduce the domain gap between omnidirectional and perspective images. The Yin-Yang grid can use the existing CNN structure as it is, but its quasi-uniform characteristic allows the decomposed image to be similar to a perspective image, so it can exploit the strong prior knowledge of the learned feed-forward network. OmniSplat demonstrates higher reconstruction accuracy than existing feed-forward networks trained on perspective images. Furthermore, we enhance the segmentation consistency between omnidirectional images by leveraging attention from the encoder of OmniSplat, providing fast and clean 3DGS editing results.

Introduction

While 3D Gaussian Splatting (3DGS) has revolutionized real-time 3D reconstruction, most "feed-forward" models—those that generate a scene instantly without per-scene optimization—are strictly limited to perspective cameras with narrow fields of view. Omnidirectional images (360° views) are ideal for capturing entire environments but pose significant challenges for deep learning. Standard equirectangular projections suffer from extreme distortion at the poles, creating a "domain gap" for neural networks trained on perspective photos. Furthermore, rendering novel views from these distorted inputs often results in "stripe-like" artifacts and holes due to non-uniform sampling.

OmniSplat addresses these issues by introducing a training-free, feed-forward framework that brings 3DGS to 360-degree omnidirectional images. By decomposing distorted spherical views into a "Yin-Yang grid," it allows models trained on standard perspective images to reconstruct holistic 3D scenes in a single pass, achieving state-of-the-art quality with sub-second inference times.

Methodology

The core innovation of OmniSplat is the use of the Yin-Yang grid to bridge the gap between spherical and perspective geometry.

Yin-Yang Decomposition and Rasterization

Instead of processing a single distorted equirectangular image, OmniSplat decomposes the sphere into two overlapping, identical pieces—the "Yin" and the "Yang"—mapped onto square grids. These grids are "quasi-uniform," meaning they exhibit significantly less distortion than standard projections. This allows the system to utilize a pre-trained perspective network (specifically MVSplat) as a feature encoder without additional fine-tuning. Unlike irregular topologies like icosahedrons, the Yin-Yang grid’s square structure makes it "plug-and-play" for existing computer vision architectures.

The OmniSplat pipeline: Yin-Yang decomposition, feature encoding, cross-view attention for 3DGS parameter estimation, and specialized rasterization.

The model takes Yin and Yang images from reference views and uses a multiview Transformer to estimate 3D Gaussian parameters in a pixel-aligned manner. For rendering, the Yin and Yang parts are processed separately using a specialized rasterizer and then merged in pixel space, preventing artifacts typically caused by high-frequency sampling at the poles.

Editable Capabilities

By leveraging the attention mechanisms used to match features across views, OmniSplat produces multiview-consistent attention scores. These scores enable fast semantic segmentation, allowing users to perform 3D scene editing, such as object removal, almost instantly.

Experimental Results

OmniSplat was evaluated across six benchmark omnidirectional datasets, including synthetic (OmniBlender) and real-world environments (Ricoh360, 360Roam, etc.).

Quantitative and Qualitative Performance

OmniSplat generates a 3D scene in just 0.532 seconds, making it approximately 3,600 times faster than optimization-based methods like ODGS. On the OmniBlender dataset, OmniSplat achieved a PSNR of 21.22, significantly outperforming feed-forward baselines like MVSplat (17.48) or PixelSplat (20.56) adapted for 360° views. Even higher quality can be achieved with "OmniSplat+opt," which uses a brief optimization step (100 iterations, ~12 seconds) to rival or exceed optimization-heavy methods that typically require over 30 minutes.

Qualitative comparison of OmniSplat against state-of-the-art methods. OmniSplat eliminates ghosting and stripe artifacts, particularly in polar regions.

Qualitatively, OmniSplat eliminates the "ghosting" and "stripe" artifacts seen in previous methods, producing much cleaner and more photo-realistic novel views. The "training-free" advantage suggests that geometric priors learned by large-scale perspective models are highly transferable when the input is appropriately reformatted.

Conclusion

OmniSplat demonstrates that feed-forward 3DGS can be effectively extended to omnidirectional images through clever geometric decomposition. By using the Yin-Yang grid, the model exploits existing perspective priors to achieve high-quality, real-time reconstruction. While current limitations include minor interpolation artifacts and a focus on static scenes, OmniSplat provides a robust foundation for future work in dynamic 360° reconstruction and efficient scene editing.

Summary updated

Abstract

Feed-forward 3D Gaussian Splatting (3DGS) models have gained significant popularity due to their ability to generate scenes immediately without needing per-scene optimization. Although omnidirectional images are getting more popular since they reduce the computation for image stitching to composite a holistic scene, existing feed-forward models are only designed for perspective images. The unique optical properties of omnidirectional images make it difficult for feature encoders to correctly understand the context of the image and make the Gaussian non-uniform in space, which hinders the image quality synthesized from novel views. We propose OmniSplat, a pioneering work for fast feed-forward 3DGS generation from a few omnidirectional images. We introduce Yin-Yang grid and decompose images based on it to reduce the domain gap between omnidirectional and perspective images. The Yin-Yang grid can use the existing CNN structure as it is, but its quasi-uniform characteristic allows the decomposed image to be similar to a perspective image, so it can exploit the strong prior knowledge of the learned feed-forward network. OmniSplat demonstrates higher reconstruction accuracy than existing feed-forward networks trained on perspective images. Furthermore, we enhance the segmentation consistency between omnidirectional images by leveraging attention from the encoder of OmniSplat, providing fast and clean 3DGS editing results.
BibTeX
@inproceedings{lee2025omnisplat,
  author = {Suyoung Lee and Jaeyoung Chung and Kihoon Kim and Jaeyoo Huh and Gunhee Lee and Minsoo Lee and Kyoung Mu Lee},
  title = {{OmniSplat: Taming Feed-Forward 3D Gaussian Splatting for Omnidirectional Images with Editable Capabilities}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=u8aC6LBquX},
}
Figure