Spherical Image Generation from a Single Image by Considering Scene Symmetry

Takayuki Hara, Yusuke Mukuta, Tatsuya HaradaAAAI 2021 · OpenReview

Abstract

Spherical images taken in all directions allow the full surroundings of a subject to be represented, providing an immersive experience. Generating a spherical image from a single normal-field-of-view (NFOV) image is convenient but challenging due to the high degree of freedom in generating unobserved regions. We propose a method that leverages scene symmetry—rotational, planar, and asymmetric—as a structural prior. By incorporating symmetry intensity as a latent variable in a Conditional Variational Autoencoder (CVAE) and using circular shifts and flips of latent features, we control the generated regions. Our results demonstrate that this approach produces plausible, seamless spherical images with reduced reconstruction errors.

Introduction

Spherical (360°) images are essential for virtual reality and immersive content, yet capturing them typically requires specialized hardware or complex stitching. Generating a full 360° environment from a single standard photograph (NFOV) would significantly lower the barrier for VR content creation. However, this is an extreme image completion task. Standard models often fail to maintain spherical continuity, leading to visible seams, and struggle to hallucinate the vast unobserved areas without a strong structural prior.

A spherical image representing a full 360° environment.

Methodology: Controlling Symmetry in Latent Space

The proposed method utilizes a Conditional Variational Autoencoder (CVAE) architecture designed for equirectangular projections. The core innovation lies in treating scene symmetry as a controllable latent variable.

Symmetry as a Structural Prior

Most real-world scenes, such as architectural interiors or courtyards, exhibit global symmetry. The model introduces a symmetry intensity vector $s$ to control five types of symmetry: 90°, 180°, and 270° rotational symmetry, and 0° and 90° plane symmetry.

The model supports various symmetry types, including rotational and planar symmetries, to guide the generation of unobserved regions.

Geometric Transformations and Circular Padding

To enforce these symmetries, the model applies explicit geometric operations—horizontal circular shifts for rotational symmetry and flips for plane symmetry—directly to the CNN feature maps. Furthermore, to ensure the generated 360° image is perfectly continuous at the boundaries, standard zero-padding is replaced with circular padding in all convolutional layers. This architectural tweak eliminates the edge discontinuities common in other equirectangular generation methods.

Experimental Results

The model was trained on the SUN360 dataset, using 50,000 images for training and 10,000 for testing. Input NFOV images were synthetically generated with fields-of-view ranging from 30° to 120°.

Quantitative Performance

The proposed method significantly outperformed state-of-the-art baselines like PICNet and 360IC. In generation tasks, it achieved a Frechet Inception Distance (FID) of 24.4 (vs. PICNet's 28.3). In reconstruction, it reached a PSNR of 14.3, compared to 12.5 for PICNet.

Qualitative Control and Seamlessness

Beyond metrics, the model allows users to manually adjust the symmetry latent variable during inference, enabling smooth transitions from asymmetric to highly symmetric outputs. Qualitative evaluations confirmed that the circular padding successfully produced seamless equirectangular projections, a key advantage over previous methods.

Conclusion

By explicitly modeling scene symmetry and adapting CNN architectures for spherical geometry, this work provides a robust framework for generating immersive 360° environments from limited visual data. While challenges remain in controlling local object placement and generalizing to highly unusual scenes, the use of symmetry as a global structural prior marks a significant step forward in unconstrained image completion.

Summary updated

Abstract

Spherical images taken in all directions (360 degrees by 180 degrees) allow the full surroundings of a subject to be represented, providing an immersive experience to viewers. Generating a spherical image from a single normal-field-of-view (NFOV) image is convenient and expands the usage scenarios considerably without relying on a specific panoramic camera or images taken from multiple directions; however, achieving such images remains a challenging and unresolved problem. The primary challenge is controlling the high degree of freedom involved in generating a wide area that includes all directions of the desired spherical image. We focus on scene symmetry, which is a basic property of the global structure of spherical images, such as rotational symmetry, plane symmetry, and asymmetry. We propose a method for generating a spherical image from a single NFOV image and controlling the degree of freedom of the generated regions using the scene symmetry. To estimate and control the scene symmetry using both a circular shift and flip of the latent image features, we incorporate the intensity of the symmetry as a latent variable into conditional variational autoencoders. Our experiments show that the proposed method can generate various plausible spherical images controlled from symmetric to asymmetric, and can reduce the reconstruction errors of the generated images based on the estimated symmetry.
BibTeX
@inproceedings{hara2021spherical,
  author = {Takayuki Hara and Yusuke Mukuta and Tatsuya Harada},
  title = {{Spherical Image Generation from a Single Image by Considering Scene Symmetry}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=NNicrtYBIHw6},
}
Figure