TL;DR
CubeComposer is the first spatio-temporal autoregressive diffusion model capable of natively generating 4K 360° panoramic videos from standard perspective video inputs. By decomposing the panorama into a six-faced cubemap and generating content face-by-face and window-by-window, it bypasses the prohibitive memory constraints of standard full-attention diffusion models without relying on artifact-prone post-processing super-resolution.
What Problem Does This Paper Address?
Creating immersive 360° panoramic videos typically requires expensive multi-camera rigs or specialized omnidirectional hardware. An appealing alternative is "perspective-to-360° video generation," which takes a standard video clip (with some camera rotation) and outpaints the rest of the environment to form a full panoramic view.
However, virtual reality (VR) applications demand at least 4K resolution (3840×1920) for a truly immersive experience. Existing video diffusion models rely on full self-attention mechanisms, which scale quadratically ($O(N^2)$) with the number of tokens. As a result, state-of-the-art models are computationally bottlenecked to a native resolution of 1K or less. To reach higher resolutions, current pipelines resort to external video super-resolution (VSR) models as a post-processing step. Because these VSR models lack intrinsic generative reasoning about the 360° scene, they often introduce severe visual artifacts and fail to generate fine, realistic details.
Methods
CubeComposer shifts away from the traditional Equirectangular Projection (ERP) format, which suffers from non-uniform distortion near the poles. Instead, it adopts a cubemap representation consisting of six faces (Front, Right, Back, Left, Up, Down). This format aligns much better with the spatial priors of foundational video diffusion models (specifically, the authors build on the Wan 2.2 5B model).
The core architecture introduces three key components to make 4K generation tractable and seamless:
- Spatio-Temporal Autoregressive Planning: The video is divided into short temporal windows. Within each window, CubeComposer determines the spatial generation order based on how much of each cube face is covered by the input perspective video. Generating the most "certain" (highest coverage) faces first provides strong geometric and appearance anchors that propagate naturally to the remaining unobserved faces.
- Context Mechanism with Sparse Attention: To maintain cross-face and temporal coherence, the model conditions each step on a mixture of historical tokens (past windows), current window tokens, and future fragments from the input perspective video. To prevent memory explosion from this massive context, the authors designed a Sparse Context Attention (SCA) mechanism. While the target generation sequence performs full self-attention, the context sequence only attends to itself via a local diagonal-banded mask, reducing the computational complexity from quadratic to linear ($O(C \cdot K)$).
- Continuity-Aware Designs: Generating faces independently can cause ugly boundary seams. CubeComposer solves this by remapping positional encodings to match a flattened cubemap topology and implementing a topology-aligned padding and blending strategy during latent generation and pixel decoding.
Datasets
To facilitate high-resolution training, the authors introduced a new dataset alongside an existing benchmark:
- 4K360Vid (New): A curated dataset of 11,832 high-quality 360° video clips at $\ge$ 4K resolution. The dataset includes global and frame-wise captions generated via the Qwen3-VL 235B model, which was also used to filter out low-quality or anomalous data.
- ODV360: A standard public 360° video dataset, from which the authors utilized the high-resolution subset for training and evaluation.
Key Insights and Ideas
- Divide and Conquer Spatially: Autoregression isn't just for the time domain. By treating spatial faces as autoregressive steps, the peak memory footprint drops to that of a single standard video frame sequence, unlocking 4K generation on consumer-accessible hardware.
- Order Matters: Causal generation in space works best when guided by information density. Sorting the face generation order based on perspective video coverage minimizes early uncertainty and prevents error accumulation.
- Future Context is Essential: Even in an autoregressive setup, incorporating sparse future "fragments" from the input video is vital for maintaining temporal consistency, especially when dealing with dynamic camera trajectories.
Key Results
CubeComposer was evaluated against leading baselines including Argus, Imagine360, and ViewPoint.
Quantitative Performance
When evaluated on the 4K360Vid dataset, CubeComposer significantly outperformed previous methods across reference-based metrics (LPIPS, CLIP) and distributional metrics (FID, FVD). Notably, on Fréchet Video Distance (FVD)—which is highly sensitive to temporal dynamics—CubeComposer achieved a score of 2.2205 at native 4K, vastly outperforming Argus (4.0755 at 1K) and Argus+VEnhancer (6.1337 at 2K).
Qualitative Performance
Qualitative comparisons show that while traditional VSR upscalers blur fine textures or introduce warping artifacts, CubeComposer generates sharp, structurally sound, and temporally stable environments.
Limitations and Future Work
While CubeComposer achieves a major milestone in resolution, the autoregressive generation of six individual faces across multiple time windows inherently increases total inference time. The authors note that future work will focus on:
- Optimizing efficiency by reducing the number of required diffusion steps.
- Exploring streaming architectures to amortize computation and lower latency, pushing the method closer to real-time interactive VR applications.
Summary updated