Conditional 360-degree Image Synthesis for Immersive Indoor Scene Decoration

Ka-Chun Shum, Hong-Wing Pang, Binh-Son Hua, Duc Thanh Nguyen, Sai-Kit Yeung

Abstract: In this paper, we address the problem of conditional scene decoration for 360° images. Our method takes a 360° background photograph of an indoor scene and generates decorated images of the same scene in the panorama view. To do this, we develop a 360-aware object layout generator that learns latent object vectors in the 360° view to enable a variety of furniture arrangements for an input 360° background image. We use this object layout to condition a generative adversarial network to synthesize images of an input scene. To further reinforce the generation capability of our model, we develop a simple yet effective scene emptier that removes the generated furniture and produces an emptied scene for our model to learn a cyclic constraint. We train the model on the Structure3D dataset and show that our model can generate diverse decorations with controllable object layout. Our method achieves state-of-the-art performance on the Structure3D dataset and generalizes well to the Zillow indoor scene dataset. Our user study confirms the immersive experiences provided by the realistic image quality and furniture layout in our generation results.

Introduction

Creating immersive virtual reality (VR) experiences often requires decorated indoor scenes. While 360-degree (equirectangular) images provide the necessary field of view, automatically generating furniture within them is difficult for three main reasons:

  1. Geometric Distortion: Standard 2D image techniques fail because objects in a 360° view warp significantly, especially near the poles.
  2. Controllability: Most generative models are "black boxes"—they produce an image without allowing for easy manipulation of specific objects like couches or tables.
  3. Data Scarcity: There isn't a single "correct" way to furnish a room, making it difficult to train models using traditional supervised learning that expects a 1:1 match between input and output.

Previous state-of-the-art methods often required users to manually draw bounding boxes for furniture, which is difficult for non-experts and defeats the purpose of automation.

Methodology

The proposed framework utilizes a three-part pipeline to address these challenges:

  • 360-Aware Object Layout Generator: Instead of using 2D rectangles, the model learns a set of "object vectors" representing furniture as ellipses in a spherical coordinate system. This handles the "wrap-around" nature of panoramas and the inherent distortions of equirectangular projections.
  • Conditional Scene Decorator: Based on the StyleGAN2 architecture, this module takes the empty background and the generated layout to synthesize the final decorated image. It uses spatial modulation to ensure the furniture fits the structure of the room.
  • The Scene Emptier (Cycle-Consistency): To train the model without needing a perfect ground-truth match for every possible furniture arrangement, the authors created a "scene emptier" that learns to remove furniture. By enforcing a cycle-consistency loss (Empty Room → Furnished → Emptied ≈ Original), the model learns to generate plausible decorations while strictly preserving the original room's architecture.
The proposed framework architecture, highlighting the Layout Generator, Scene Decorator, and the Scene Emptier for cycle-consistency.

To ensure the left and right edges of the panorama match up seamlessly, the authors implemented horizontal circular padding in all convolutional layers, preventing visible seams in the generated 360° view.

Datasets and Insights

The researchers utilized two primary datasets: Structured3D, a large-scale synthetic dataset containing over 21,000 pairs of furnished and unfurnished scans, and the Zillow Indoor Dataset (ZInD), a real-world dataset used to prove the model's generalization to actual photography.

Key insights from the development include:

  • Geodesic Logic: Modeling objects as ellipses in spherical space allows the model to maintain object integrity across the entire panorama.
  • Unsupervised Layout Learning: The layout generator identifies "slots" where objects could exist based on the floor plan and walls, rather than requiring explicit labels.
  • The Emptier as a Critic: The scene emptier forces the generator to produce clean, structurally sound furniture; if the decorator generates a "messy" image that cannot be reverted, the model is penalized.

Results

The model achieved state-of-the-art scores in Frechet Inception Distance (FID) and Kernel Inception Distance (KID). On the bedroom subset, it achieved an FID of 64.55, significantly outperforming baselines like Pix2PixHD (68.97).

Qualitative comparison of the proposed method against state-of-the-art baselines, demonstrating superior realism and structural consistency.

In a user study with 35 participants, the proposed method was consistently ranked first for both photo-realism and the quality of furniture arrangement. Furthermore, because the layout is represented by latent vectors, users can manually "nudge" the ellipses to move, resize, or remove furniture in real-time.

Conclusion and Future Work

While the results are state-of-the-art, the authors identify areas for future improvement. The model currently lacks semantic awareness (e.g., distinguishing a bed from a sofa), which could be addressed by incorporating semantic labels. Additionally, integrating deeper 3D structural understanding, such as depth maps or point clouds, could lead to even more precise furniture placement in complex architectural environments.

Summary updated

Abstract

In this paper, we address the problem of conditional scene decoration for 360° images. Our method takes a 360° background photograph of an indoor scene and generates decorated images of the same scene in the panorama view. To do this, we develop a 360-aware object layout generator that learns latent object vectors in the 360° view to enable a variety of furniture arrangements for an input 360° background image. We use this object layout to condition a generative adversarial network to synthesize images of an input scene. To further reinforce the generation capability of our model, we develop a simple yet effective scene emptier that removes the generated furniture and produces an emptied scene for our model to learn a cyclic constraint. We train the model on the Structure3D dataset and show that our model can generate diverse decorations with controllable object layout. Our method achieves state-of-the-art performance on the Structure3D dataset and generalizes well to the Zillow indoor scene dataset. Our user study confirms the immersive experiences provided by the realistic image quality and furniture layout in our generation results. Our implementation is available at https://github.com/kcshum/neural_360_decoration.git.
BibTeX
@inproceedings{shum2023conditional,
  author = {Ka-Chun Shum and Hong-Wing Pang and Binh-Son Hua and Duc Thanh Nguyen and Sai-Kit Yeung},
  title = {{Conditional 360-degree Image Synthesis for Immersive Indoor Scene Decoration}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=wNEKyRu2q3},
}
Figure