Introduction
Generating high-quality 360-degree panoramas from text descriptions presents a significant challenge for standard diffusion models. While Stable Diffusion excels at creating perspective images, it struggles with the equirectangular projection required for panoramas. This "domain gap" is compounded by data scarcity; high-quality panoramic datasets like Matterport3D are orders of magnitude smaller than their perspective counterparts.
Previous state-of-the-art (SOTA) methods attempted to solve this using complex "dual-branch" architectures that processed perspective and panoramic views separately. However, these models are computationally intensive and difficult to scale. This research investigates the internal mechanisms of Stable Diffusion during fine-tuning to develop a more efficient, single-branch solution called UniPano.
Methodology: A Surgical Investigation
The researchers conducted a detailed analysis of Low-Rank Adaptation (LoRA) within the cross-attention layers of Stable Diffusion. By decomposing these layers into four trainable components—Query ($W_q$), Key ($W_k$), Value ($W_v$), and Output ($W_o$)—they sought to identify which parts of the model are responsible for learning spherical distortions.
Based on this investigation, the authors introduced UniPano, a uni-branch framework that:
- Freezes $W_q$ and $W_k$: Since these matrices primarily preserve general perspective knowledge, they do not need extensive fine-tuning for panoramic structures.
- Employs Mixture of Experts (MoE): The LoRA in the $W_o$ layer is replaced with an MoE module, allowing "specialist" weights to handle complex distortions without a massive increase in active parameters.
The Functional Division of Attention
The study revealed a surprising "division of labor" within the attention blocks. When trained in isolation, the $W_q$ and $W_k$ matrices fail to generate the characteristic "curvy" lines of a panorama. In contrast, $W_v$ and $W_o$ successfully adapt to the panoramic domain.
This suggests that $W_{v,o}$ act as the primary "adapters" for structural transformation, while $W_{q,k}$ serve as "preservers" of the model's original generative capabilities. Interestingly, removing the $W_{v,o}$ LoRAs after training allows the model to revert to generating standard perspective images, proving that the core knowledge remains intact.
Results and Efficiency
UniPano achieves state-of-the-art performance while significantly reducing resource requirements:
- Quality: It reached a 5.90 FAED score, surpassing the previous leader, PanFusion (6.04).
- Memory: Training requires only 32.59 GB of VRAM, a nearly 50% reduction from PanFusion's 60.12 GB.
- Speed: Training is approximately twice as fast, completing a 10-epoch cycle in 3.4 hours.
- Scalability: The framework successfully generates high-resolution 1024x2048 panoramas using Stable Diffusion 3.
Conclusion
By identifying the specific roles of attention matrices, the authors have simplified the task of panorama generation. UniPano demonstrates that a complex dual-branch setup is unnecessary, paving the way for high-resolution panoramic generation on consumer-grade hardware. Future work will focus on improving layout logic to prevent "impossible" architectural configurations and optimizing the Mixture of Experts hyperparameters.
Summary updated