TL;DR
PanoEvo introduces a "panoramic world state" to solve the persistent problem of objects freezing or vanishing when a video model's camera turns away. By evolving the entire 360° scene in a panoramic latent space and using a dedicated perspective renderer to generate high-quality views, it achieves state-of-the-art performance in maintaining world consistency and object evolution.
What Problem Does This Paper Address?
Imagine a video model where a man is asked to walk to a sofa and sit down. If the virtual camera pans 60° away while he’s walking and then pans back, most current models fail. They might show the man still standing where he was, or worse, he might have vanished or turned into a different person entirely.
This "out-of-sight, out-of-mind" issue stems from the fact that standard perspective video models only represent what is currently in the frame. They lack a persistent representation of the world outside the field of view. While some researchers have tried using 3D memories or geometry caches, these often treat the unobserved world as static. PanoEvo tackles the challenge of dynamic persistence: ensuring the world keeps evolving realistically even when you aren't looking at it.
Methods
The authors propose a factorized approach that splits world evolution from view rendering:
- Panoramic Evolver ($\mathcal{E}$): This model takes a single initial image and a text prompt to generate an evolving panoramic latent state ($W$). Built on a panoramic video diffusion transformer, it represents the entire 360° world at every timestep. This ensures that if a person starts walking in the prompt, they continue walking in the latent space, regardless of where the camera points.
- Perspective Renderer ($\mathcal{R}$): Directly projecting a low-resolution panorama into a perspective view often results in blurry details. To fix this, the authors built a renderer based on a high-quality perspective video model (Wan2.1). It "reads" the panoramic state through two pathways:
- Projected Latent Maps: A pixel-aligned estimate that anchors the geometry.
- Frustum Cross-Attention: A mechanism allowing video tokens to attend to panoramic context within a specific angular radius (15°), ensuring the renderer stays faithful to the evolved world state while adding sharp, high-resolution details.
Datasets
Training a model to evolve a full 360° scene from a single perspective image required a new kind of data. The authors introduced PanoEvo-Data, curated from real 360° videos:
- Coarse Subset (20,812 clips): Used for a "Completion stage" to teach the model how to fill in the rest of the panorama from a single view.
- Event Subset (1,400 samples): Highly curated clips with verified object-level actions (walking, sitting, interacting) used for the "Evolution stage."
- Render Subset (20,526 clips): High-resolution source views used to train the Perspective Renderer to produce sharp output.
Key Insights and Ideas
- Latent Persistence is Key: Instead of explicit 3D modeling, a panoramic latent space provides a dense, continuous representation of the world. It’s a "cheat code" for consistency because every direction is always present.
- Coarse-to-Fine Training: The model first learns to "complete" a static panorama from a slice, then learns how objects within that panorama "evolve" over time. This staged training is crucial for handling the massive search space of 360° video.
- The Renderer as a "Smart Upsampler": The Perspective Renderer doesn't just copy the panorama; it treats it as a noisy estimate to be refined. This allows it to correct structural errors or blurriness inherent in equirectangular projections.
Key Results
PanoEvo was evaluated on two major benchmarks for world models:
- WRBench: It achieved the best overall score (.753) among 23 publicly evaluated models, significantly outperforming the strongest baselines in re-observation metrics (how well the model remembers a scene after looking away).
- STEVO-Bench: It attained the highest total score (364.7 vs. 300.9 for the previous leader, ReMind), showing superior state progress and physical plausibility.
- Qualitative Success: As seen in Figure 4, while baselines often lose the subject or freeze their state, PanoEvo successfully completes actions like sitting down or placing an object while the camera is turned away.
Limitations and Future Work
- Fixed Camera Center: Currently, PanoEvo only handles camera rotation. It does not account for translation (moving the camera through space), which would introduce complex parallax effects that a single-center panorama cannot represent.
- Data Diversity: Real-world 360° footage of certain physical processes (like liquids or melting) is rare, meaning the model is less proficient at evolving these specific types of scenes.
- Computational Overhead: Running two separate diffusion models is computationally intensive, though the authors note that the panoramic state only needs to be generated once to render any number of camera trajectories.
Summary updated