TL;DR
Pantheon360 is a novel framework that generates temporally and geometrically consistent 360° videos from sparse or single panoramic images along any user-defined camera trajectory. By decoupling geometric structure (via an explicit 3D point cloud cache) from photorealistic texture synthesis (via a fine-tuned video diffusion model), it overcomes the view-inconsistency and drift limitations of standard perspective video generators.
What Problem Does This Paper Address?
Building digital twins for robotics, autonomous driving, and simulation requires generating complete, photorealistic 3D worlds from video or image inputs with precise camera control. Current video generation models mostly operate on standard perspective images with a narrow Field of View (FoV). When simulating long or complex trajectories, these models are "blind" to what lies outside the initial frame, forcing them to repeatedly guess and hallucinate unseen regions. This leads to severe spatial-temporal inconsistencies—for instance, an object might completely change its appearance when the camera loops back to view it from another angle.
While 360° video generation provides full global context from the start, it introduces extreme equirectangular distortion and makes precise geometric camera control highly challenging. Existing 360° models are either limited to simple high-level actions (like "move forward") or only work on synthetic data, failing to handle complex, real-world scenes.
Methods
Pantheon360 introduces a smart division of labor: an explicit 3D Cache handles the complex 3D geometric reasoning, while a fine-tuned Latent Video Diffusion Model (built on Stable Video Diffusion) focuses purely on photorealistic texture refinement.
- 3D Cache Reconstruction: At inference time, the sparse 360° input images are cropped into multiple overlapping perspective views. These views are processed by 3D foundation models (like $\pi^3$ or VGGT) to generate a unified 3D point cloud representation of the scene.
- Geometric Conditioning ($V_{geo}$): The 3D point cloud is rendered along the user's target camera trajectory into an equirectangular projection (ERP) video. This geometry-only video ($V_{geo}$) acts as a structural scaffold. It is encoded into the latent space and concatenated with the noised latents at each diffusion step to enforce strict geometric consistency.
- Semantic Conditioning: To preserve visual details, the first frame is cropped into 8 perspective views, passed through a CLIP image encoder, and injected into the denoising U-Net via cross-attention.
- Dual-Anchor Latent Fusion: For video interpolation between distant frames (e.g., stitching Google Street View panoramas), the framework renders geometry forward from the start frame and backward from the end frame, smoothly blending the two streams in the latent space to prevent sudden jumps or discontinuities.
Datasets
- 360-1M: A massive dataset of real-world 360° videos. The authors applied a rigorous filtering pipeline (removing static posters, cuts, and low-motion clips) to curate a high-quality subset of 55,000 5-second clips for training. Camera trajectories and 3D caches were auto-annotated on-the-fly using ViPE.
- Web360: Used to evaluate single-view to video generation in diverse, real-world outdoor environments.
- Habitat: A synthetic indoor dataset used to evaluate sparse-view video generation along complex, non-looped polyline trajectories.
- Google Maps Street View: Used to demonstrate real-world downstream applications like novel view synthesis and motion interpolation.
Key Insights and Ideas
- Decoupling Structure from Texture: Grounding the generation in an explicit 3D point cloud scaffold ensures strict geometric consistency across viewpoints, freeing the diffusion model to focus entirely on photorealism.
- Panoramic Context Eliminates Drift: 360° inputs capture the entire scene context at $t=0$. This global awareness simplifies camera trajectory design and prevents the cross-view drift common in perspective models.
- Perspective Cropping for Foundation Models: Standard 3D foundation models and CLIP struggle with distorted equirectangular images. Cropping the 360° image into overlapping perspective views allows Pantheon360 to leverage powerful, pre-trained perspective models seamlessly.
Key Results
Pantheon360 significantly outperforms perspective-based state-of-the-art baselines (ViewCrafter, TrajectoryCrafter, and GEN3C) adapted for 360° tasks.
- Single-View Generation (Web360): Achieved an FVD of 356.15 (vs. 380.08 for the best baseline, GEN3C) and a massive jump in structural consistency (SSIM of 0.746 vs. 0.583).
- Sparse-View Generation (Habitat): Outperformed baselines across all metrics, notably reducing the 3D geometric consistency error (MET3R score of 0.3026 vs. 0.4522 for GEN3C).
- Downstream Applications: Successfully enabled practical use cases like 360° video stabilization (smoothing shaky footage) and long-range motion interpolation using Google Street View data.
Limitations and Future Work
- Dynamic Objects: The 3D Cache primarily encodes static scene geometry. While the diffusion model's priors can handle minor dynamic elements, explicit control over object-level dynamics (e.g., moving cars or pedestrians) remains a challenge.
- Artifact Propagation: If the input 360° images contain pre-existing stitching artifacts, or if the scene is highly crowded and causes motion blur, these errors can propagate through the pipeline.
- Future Directions: The authors suggest incorporating explicit 4D motion representations in future work to enable fine-grained control over dynamic object behavior.
Summary updated