What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?

Jing Zhang (张静), Jinhong Ni, Chang-Bin Zhang, Qiang Zhang, Jing Zhang

Introduction

Generating high-quality 360-degree panoramas from text descriptions presents a significant challenge for standard diffusion models. While Stable Diffusion excels at creating perspective images, it struggles with the equirectangular projection required for panoramas. This "domain gap" is compounded by data scarcity; high-quality panoramic datasets like Matterport3D are orders of magnitude smaller than their perspective counterparts.

Previous state-of-the-art (SOTA) methods attempted to solve this using complex "dual-branch" architectures that processed perspective and panoramic views separately. However, these models are computationally intensive and difficult to scale. This research investigates the internal mechanisms of Stable Diffusion during fine-tuning to develop a more efficient, single-branch solution called UniPano.

Methodology: A Surgical Investigation

The researchers conducted a detailed analysis of Low-Rank Adaptation (LoRA) within the cross-attention layers of Stable Diffusion. By decomposing these layers into four trainable components—Query ($W_q$), Key ($W_k$), Value ($W_v$), and Output ($W_o$)—they sought to identify which parts of the model are responsible for learning spherical distortions.

Architecture of the Multi-head Cross-Attention (MCA) block showing the placement of LoRA adapters on the linear projection matrices.

Based on this investigation, the authors introduced UniPano, a uni-branch framework that:

  1. Freezes $W_q$ and $W_k$: Since these matrices primarily preserve general perspective knowledge, they do not need extensive fine-tuning for panoramic structures.
  2. Employs Mixture of Experts (MoE): The LoRA in the $W_o$ layer is replaced with an MoE module, allowing "specialist" weights to handle complex distortions without a massive increase in active parameters.

The Functional Division of Attention

The study revealed a surprising "division of labor" within the attention blocks. When trained in isolation, the $W_q$ and $W_k$ matrices fail to generate the characteristic "curvy" lines of a panorama. In contrast, $W_v$ and $W_o$ successfully adapt to the panoramic domain.

Visual comparison of panorama generation results when training different attention components. The left image (a) shows a failure to capture spherical geometry compared to the successful adaptation in the right image (b).

This suggests that $W_{v,o}$ act as the primary "adapters" for structural transformation, while $W_{q,k}$ serve as "preservers" of the model's original generative capabilities. Interestingly, removing the $W_{v,o}$ LoRAs after training allows the model to revert to generating standard perspective images, proving that the core knowledge remains intact.

Results and Efficiency

UniPano achieves state-of-the-art performance while significantly reducing resource requirements:

  • Quality: It reached a 5.90 FAED score, surpassing the previous leader, PanFusion (6.04).
  • Memory: Training requires only 32.59 GB of VRAM, a nearly 50% reduction from PanFusion's 60.12 GB.
  • Speed: Training is approximately twice as fast, completing a 10-epoch cycle in 3.4 hours.
  • Scalability: The framework successfully generates high-resolution 1024x2048 panoramas using Stable Diffusion 3.

Conclusion

By identifying the specific roles of attention matrices, the authors have simplified the task of panorama generation. UniPano demonstrates that a complex dual-branch setup is unnecessary, paving the way for high-resolution panoramic generation on consumer-grade hardware. Future work will focus on improving layout logic to prevent "impossible" architectural configurations and optimizing the Mixture of Experts hyperparameters.

Summary updated

BibTeX
@inproceedings{2025what,
  author = {Jing Zhang (张静) and Jinhong Ni and Chang-Bin Zhang and Qiang Zhang and Jing Zhang},
  title = {{What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=PSMZwFmqUb},
}
Figure