Autoregressive View Model

Jensen, Zhou, Samuel Rota Bulo, Denis Rozumny, Duncan Zauss, Peter Hedman, Nikhil Keetha, Philip Torr, Christian Rupprecht, Andrea Vedaldi, Peter KontschiederarXiv 2026 · arXiv

TL;DR

The Autoregressive View Model (AVM) generates complex 3D scenes as a continuous stream of views without ever building an explicit 3D representation like a point cloud or voxel grid. By treating scene generation as an in-context sequence task and using a novel retrieval mechanism to maintain long-term memory, AVM achieves state-of-the-art consistency and camera control across static and dynamic environments.

What Problem Does This Paper Address?

Creating a consistent video of a 3D scene from just a few images is a "holy grail" in computer vision. Current approaches generally fall into two camps:

  1. Reconstruction methods (like NeRF or 3D Gaussian Splatting) produce highly consistent results but require dense input data and time-consuming per-scene optimization.
  2. Video generation models can "hallucinate" new content from a single image but often suffer from "drift"—as the camera moves further away, the model forgets what it previously generated, leading to broken geometry and inconsistent textures when the camera returns to a known spot.

AVM seeks a middle ground: a model that can reconstruct, explore, and revisit scenes with the flexibility of a generative model but the persistence of a 3D representation.

Methods

AVM is built on a 14-billion parameter video diffusion transformer. Unlike standard video models that generate a fixed-length clip, AVM uses an $N$-in-1-out formulation: it conditions on $N$ context slots to generate one new target clip.

The core technical innovation lies in how it manages "memory" through a unified in-context interface:

  • Context Management: The model doesn't just look at the last few frames. It uses a bounded active context consisting of "Sink" views (initial anchor images), "Retrieved" views (past views spatially near the current camera), and "Motion" context (the most recent frames).
  • Retrieval Policy: To handle long trajectories, AVM retrieves past views from a history pool based on camera frustum overlap. This allows the model to "remember" a room it saw minutes ago when the camera loops back.
  • Causal Tuning: The authors first train a bidirectional model, then fine-tune it using "causal tuning" on its own rollouts. This stage uses Error Recycling—intentionally exposing the model to its own past mistakes during training—to make it robust against the compounding errors typical of autoregressive generation.
The AVM architecture treats a scene as a stream of views, using prefill and history as context to generate target views along a camera path.

Datasets

The model was trained at a massive scale on approximately 8 million scenes and video clips. This includes:

  • Labeled Scenes: Indoor (ScanNet++, RealEstate10K), outdoor (DL3DV), and synthetic environments (Aria, ParallelDomain-4D).
  • Unlabeled Videos: "In-the-wild" clips to help the model learn natural dynamics and text-prompt following.
  • Benchmarks: Evaluation was performed on DL3DV-Benchmark, WorldScore, and MultiCam, covering camera control, visual quality, and 4D reconstruction.

Key Insights and Ideas

  • 3D is a Sequence Problem: The most striking insight is that explicit 3D geometry (voxels, meshes) might be unnecessary for many applications if you have a strong enough video prior and a smart way to retrieve relevant "history."
  • Retrieval as a Differentiable Memory: By explicitly retrieving past views based on camera poses (as seen in Figure 4), the model maintains 3D consistency over long horizons that would overwhelm a standard transformer's attention window.
  • The Power of "Error Recycling": Autoregressive models usually fail because they aren't trained on their own "hallucinations." By feeding the model's errors back into the training loop, the authors significantly reduced long-term drift.
Context management allows AVM to assemble a bounded context from a large pool of history, ensuring the model stays grounded in past observations.

Key Results

AVM demonstrates remarkable versatility, handling everything from single-image animation to sparse 3D reconstruction within the same interface.

  • Camera Controllability: On the DL3DV benchmark, AVM achieved a camera control score of 96.48, vastly outperforming existing "world models" like LingBot-World (~58) and DreamX-World (~61).
  • Long-Horizon Consistency: In "loop closure" tests (where the camera returns to its starting point), AVM maintained a PSNR of 28.04, compared to just 12.29 for the next best baseline. This highlights the effectiveness of the retrieval mechanism.
  • Visual Quality: The model produces high-fidelity results (FID of 18.69) and successfully handles dynamic scenes, such as moving animals or people, while maintaining the background structure.
AVM handles diverse tasks including camera control, dynamics, and long-horizon consistency through a single interface.

Limitations and Future Work

Despite its strengths, the authors note several areas for improvement:

  • Text Alignment: While AVM follows camera paths perfectly, its adherence to complex text prompts (e.g., "change the room to a garden party") is currently weaker than some specialized world models.
  • Dynamic Degree: The model sometimes leans toward being "too static," producing videos with subtle motion rather than high-energy dynamics.
  • Future Directions: The team suggests integrating stronger open-weight video priors (like MiniMax H3) and adding optional geometric proxies (like depth maps) to provide even finer control over scene structure.

Summary updated

Abstract

We present the Autoregressive View Model (AVM), which generates a scene as a stream of views without constructing an explicit 3D representation. Through a unified in-context interface, AVM conditions on images or videos to generate short clips along a requested camera path. Visual context grounds reconstruction and interpolation, while a learned prior enables extrapolation into unseen regions. AVM retrieves relevant views from inputs and generated history to form a bounded active context. We fine-tune a pretrained bidirectional video diffusion transformer with flow matching in an N-in-1-out formulation. We then tune a causal model on its own rollouts to improve photorealism, reduce denoising steps, and limit long-horizon drift. We scale training to approximately 8M scenes and video clips. We evaluate generation capacity and scene persistence across benchmarks for camera control, text control, visual quality, dynamics, sparse and 4D reconstruction, and long-horizon consistency. The results demonstrate that AVM adapts to diverse input and output settings through one in-context interface.
BibTeX
@misc{jensen2026autoregressive,
  author = {Jensen and Zhou and Samuel Rota Bulo and Denis Rozumny and Duncan Zauss and Peter Hedman and Nikhil Keetha and Philip Torr and Christian Rupprecht and Andrea Vedaldi and Peter Kontschieder},
  title = {{Autoregressive View Model}},
  year = {2026},
  eprint = {2611.23417},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23417},
}
Figure