When tasked with generating a 3D object or a 360-degree panorama, standard 2D diffusion models face a fundamental limitation: they lack an inherent understanding of 3D geometry. If you use a 2D model to generate multiple viewpoints of a scene independently, the results will inevitably clash. A chair might have four legs from the front but three from the side, or an object might suffer from the "Janus problem"—the hallucination of multiple front faces across different camera angles, as described by Jiang et al.. Similarly, in human digitization, independent generation causes a person's pose or clothing to shift unnaturally as the camera rotates (Chan et al.).
To enforce 3D consistency without relying on heavy, explicit 3D representations during generation, a prominent research direction treats view synthesis as a sequential, autoregressive process. Instead of generating all views at once or completely independently, these methods generate views one after another, conditioning each new frame on the visual details established by previous frames.
The Intuition of Sequential Conditioning
The core intuition behind autoregressive multi-view generation is simple: respect the established world-state. When a generative model creates a novel viewpoint, it must "hallucinate" unobserved regions of a scene. If the model is forced to look at its own prior hallucinations, it will naturally anchor new details to the existing context.
As Spiegl et al. demonstrate, feeding previously generated frames back into the model as conditioning inputs preserves scene identity and structural continuity across extended camera paths without requiring explicit 3D geometric modeling. This sequential prior-guidance acts as a bridge across viewpoints.
This concept manifests in several distinct ways depending on the task:
- Object and Human Synthesis: Views are generated at fixed intervals around an object. The model carries over information from adjacent, previously synthesized views to ensure that textures and geometry remain aligned (AlBahar et al.; Chan et al.).
- Panoramic Environment Generation: To create a seamless 360-degree scene, models use recursive outpainting. An initial anchor view is generated, and adjacent segments are produced iteratively by conditioning the model on the overlapping edge of the preceding view (Li et al.; Lu et al.).
- Scene Navigation and Inpainting: When navigating a 3D environment, a recursive inpainting process conditions each new frame on the previously inpainted frame, minimizing flickering and texture drift as the camera moves (Zhang et al.).
Passing the Torch: Conditioning Strategies
Conditioning frame $t$ on all previous frames ($1$ to $t-1$) quickly becomes computationally intractable. Researchers have developed several clever strategies to pass context forward efficiently.
Stochastic Conditioning
To bypass the computational cost of conditioning on an ever-growing set of prior views, Watson introduced stochastic conditioning in the 3DiM framework. During each denoising step of the generation process, the model randomly selects just a single previously generated frame as its conditioning signal. This stochastic switching forces the entire sequence to remain bound to a shared 3D structure without requiring the model to attend to all prior views simultaneously.
Geometric Warping and Spatial Priors
A purely image-based feedback loop can still suffer from spatial drift because the model does not know where pixels should move in 3D space. To fix this, several approaches incorporate geometric constraints directly into the conditioning loop:
- Depth-Based Warping: Wynn et al. use depth information to explicitly warp previously generated frames into the current camera's perspective. These warped representations serve as explicit spatial priors that guide the generation of the next frame.
- Information Aggregation: Xiang frames 3D generation as a sequential image-to-image translation task. Instead of relying only on the immediate predecessor, the model aggregates information from all previously generated views using a weighted sum of warped frames, prioritizing visible and undistorted pixels to prevent long-term geometric drift.
- Epipolar Attention: Ye et al. integrate classical epipolar geometry constraints into the attention layers of the diffusion model, treating previously synthesized frames as explicit reference points to stabilize textures across a trajectory.
Hierarchical Conditioning
To prevent early errors from ruining a long sequence, Ye et al. organize the sequential generation into a structured hierarchy. Subsequent frames are conditioned on specific, strategically chosen prior viewpoints rather than just a linear chain, anchoring the entire 360-degree rotation firmly to the initial reference frame.
The Fundamental Trade-Off: Scalability vs. Drift
Why choose an autoregressive approach over generating all views simultaneously? The primary answer is scalability.
As Chan et al. and Höllein et al. point out, simultaneous multi-view generation models are fundamentally constrained by the fixed batch sizes used during training due to GPU memory limits. Autoregressive generation breaks this bottleneck, allowing for the creation of arbitrarily long camera trajectories and smooth 360-degree orbits by simply extending the generation chain.
However, this flexibility comes with a major catch: compounding error and geometric drift. Because each step relies on the hallucinations of the last, minor artifacts accumulate over time.
This vulnerability highlights a clear tension in the literature between sequential and simultaneous generation:
Sequential Generation (Autoregressive)
[Frame 1] ---> [Frame 2] ---> [Frame 3] ---> [Frame 4] ... (Infinite Trajectory)
* Pros: Scales to arbitrary lengths, low memory footprint per step.
* Cons: Errors compound; fails to close loops cleanly (e.g., returning to Frame 1).
Simultaneous Generation (Parallel Batch)
[Frame 1, Frame 2, Frame 3, Frame 4] generated together via shared attention.
* Pros: Perfect global coherence, zero drift, clean loop closures.
* Cons: Strictly limited by hardware memory and fixed training batch sizes.
Yu et al. note that autoregressive models frequently fail on circular trajectories because they cannot easily reconcile accumulated drift when the camera returns to its starting point. They argue that generating views simultaneously as a set is required to eliminate this drift. Similarly, Cao et al. and Zhang et al. show that generating a high volume of views or an entire procedural sequence in a single forward pass maintains better structural integrity and global style consistency than iterative methods.
To find a middle ground, researchers are working to formalize the mathematics of sequential blending. For instance, Lee et al. introduced SyncSDE, a framework based on Stochastic Differential Equations that mathematically synchronizes and stitches multiple diffusion trajectories together. This approach aims to provide the global coherence of simultaneous generation while retaining the flexibility of sequential conditioning.
Key Foundational Work
3D-TOGO: Towards Text-Guided Cross-Category 3D Object Generation (2022) — contributed 2 concept elements to this direction.
ReStGAN: A step towards visually guided shopper experience via text-to-image synthesis (2020) — contributed 1 concept elements to this direction.