TL;DR
The Autoregressive View Model (AVM) generates complex 3D scenes as a continuous stream of views without ever building an explicit 3D representation like a point cloud or voxel grid. By treating scene generation as an in-context sequence task and using a novel retrieval mechanism to maintain long-term memory, AVM achieves state-of-the-art consistency and camera control across static and dynamic environments.
What Problem Does This Paper Address?
Creating a consistent video of a 3D scene from just a few images is a "holy grail" in computer vision. Current approaches generally fall into two camps:
- Reconstruction methods (like NeRF or 3D Gaussian Splatting) produce highly consistent results but require dense input data and time-consuming per-scene optimization.
- Video generation models can "hallucinate" new content from a single image but often suffer from "drift"—as the camera moves further away, the model forgets what it previously generated, leading to broken geometry and inconsistent textures when the camera returns to a known spot.
AVM seeks a middle ground: a model that can reconstruct, explore, and revisit scenes with the flexibility of a generative model but the persistence of a 3D representation.
Methods
AVM is built on a 14-billion parameter video diffusion transformer. Unlike standard video models that generate a fixed-length clip, AVM uses an $N$-in-1-out formulation: it conditions on $N$ context slots to generate one new target clip.
The core technical innovation lies in how it manages "memory" through a unified in-context interface:
- Context Management: The model doesn't just look at the last few frames. It uses a bounded active context consisting of "Sink" views (initial anchor images), "Retrieved" views (past views spatially near the current camera), and "Motion" context (the most recent frames).
- Retrieval Policy: To handle long trajectories, AVM retrieves past views from a history pool based on camera frustum overlap. This allows the model to "remember" a room it saw minutes ago when the camera loops back.
- Causal Tuning: The authors first train a bidirectional model, then fine-tune it using "causal tuning" on its own rollouts. This stage uses Error Recycling—intentionally exposing the model to its own past mistakes during training—to make it robust against the compounding errors typical of autoregressive generation.
Datasets
The model was trained at a massive scale on approximately 8 million scenes and video clips. This includes:
- Labeled Scenes: Indoor (ScanNet++, RealEstate10K), outdoor (DL3DV), and synthetic environments (Aria, ParallelDomain-4D).
- Unlabeled Videos: "In-the-wild" clips to help the model learn natural dynamics and text-prompt following.
- Benchmarks: Evaluation was performed on DL3DV-Benchmark, WorldScore, and MultiCam, covering camera control, visual quality, and 4D reconstruction.
Key Insights and Ideas
- 3D is a Sequence Problem: The most striking insight is that explicit 3D geometry (voxels, meshes) might be unnecessary for many applications if you have a strong enough video prior and a smart way to retrieve relevant "history."
- Retrieval as a Differentiable Memory: By explicitly retrieving past views based on camera poses (as seen in Figure 4), the model maintains 3D consistency over long horizons that would overwhelm a standard transformer's attention window.
- The Power of "Error Recycling": Autoregressive models usually fail because they aren't trained on their own "hallucinations." By feeding the model's errors back into the training loop, the authors significantly reduced long-term drift.
Key Results
AVM demonstrates remarkable versatility, handling everything from single-image animation to sparse 3D reconstruction within the same interface.
- Camera Controllability: On the DL3DV benchmark, AVM achieved a camera control score of 96.48, vastly outperforming existing "world models" like LingBot-World (~58) and DreamX-World (~61).
- Long-Horizon Consistency: In "loop closure" tests (where the camera returns to its starting point), AVM maintained a PSNR of 28.04, compared to just 12.29 for the next best baseline. This highlights the effectiveness of the retrieval mechanism.
- Visual Quality: The model produces high-fidelity results (FID of 18.69) and successfully handles dynamic scenes, such as moving animals or people, while maintaining the background structure.
Limitations and Future Work
Despite its strengths, the authors note several areas for improvement:
- Text Alignment: While AVM follows camera paths perfectly, its adherence to complex text prompts (e.g., "change the room to a garden party") is currently weaker than some specialized world models.
- Dynamic Degree: The model sometimes leans toward being "too static," producing videos with subtle motion rather than high-energy dynamics.
- Future Directions: The team suggests integrating stronger open-weight video priors (like MiniMax H3) and adding optional geometric proxies (like depth maps) to provide even finer control over scene structure.
Summary updated