When generating 3D objects or scenes from a single image or text prompt, the most persistent challenge is maintaining multi-view consistency. Traditional 2D image diffusion models excel at creating realistic textures, but they lack an inherent understanding of 3D space. When prompted to generate different views of an object independently, they suffer from the "Janus problem"—producing contradictory geometry, such as an animal with multiple faces.
To solve this, a powerful paradigm has emerged: repurposing pre-trained 2D video diffusion models as geometric priors for 3D and 4D synthesis. Instead of building complex 3D-aware architectures from scratch, this approach maps spatial viewpoint changes directly onto the temporal axis of a video model.
The Geometry Hidden in Video Time
Why does a model trained to predict the next frame of a video understand the 3D geometry of a static object?
The answer lies in how video models minimize their training loss. To generate a realistic video of a camera moving through an environment, the model must learn the underlying physical regularities of our world—such as light transport, perspective shifts, and occlusions. As Daniel Watson et al. demonstrate in Controlling Space and Time with Diffusion Models, large-scale video data acts as a powerful spatiotemporal regularizer. It forces the network to develop an implicit geometric understanding to keep objects stable over time.
By treating camera movement around a 3D object as a temporal sequence, researchers can unlock these latent geometric priors. For instance, Gene Chou shows in Generating 3D-Consistent Videos from Unposed Internet Photos that video foundation models can be fine-tuned to generate spatially coherent transitions between wide-baseline images, activating 3D priors that traditional image interpolation methods lack.
Mapping Camera Trajectories to the Temporal Axis
The core mechanism of this research direction is "orbital video generation." To synthesize a 3D object, the system generates a video where the virtual camera rotates around the subject.
Spatial Viewpoint (θ) ───► 0° 90° 180° 270°
│ │ │ │
Temporal Frame (t) ───► t=1 t=2 t=3 t=4
In this formulation, the model's internal temporal attention layers and 1D convolutions—originally designed to ensure smooth motion over time—are repurposed to handle spatial transitions between camera viewpoints. Norman Müller et al. in MultiDiff: Consistent Novel View Synthesis from a Single Image and Qi Zuo et al. in VideoMV: Consistent Multi-View Generation Based on Large Video Generative Model demonstrate that these temporal attention mechanisms act as cross-view correspondence trackers. They ensure that a feature generated at a 0° viewpoint is correctly tracked, transformed, and preserved when viewed from 90° or 180°.
This strategy is widely utilized across different frameworks:
- High-Resolution Generation: Haibo Yang et al. use this orbital framing in Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models to maintain spatial consistency at high resolutions, mitigating the Janus problem by treating multi-view synthesis as a continuous sequence.
- Fast 4D Synthesis: Hanwen Liang et al. extend this in Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models to learn spatial geometry and temporal dynamics simultaneously by orbiting a moving object.
- Synthetic Data Engines: Because high-quality 3D data is scarce, Junlin Han et al. introduce VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models, which uses video diffusion models to generate millions of multi-view synthetic videos to train feed-forward 3D reconstruction models.
Sparse Reconstruction as Video Completion
Beyond generating objects from scratch, this temporal-to-spatial mapping redefines how we approach sparse-view 3D reconstruction. Traditionally, reconstructing a 3D scene from only two or three images is an ill-posed problem with severe geometric ambiguities.
By leveraging video models, researchers reframe sparse reconstruction as a video completion or inpainting task.
Fangfu Liu et al. in ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model and Yan Xu et al. in Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion show that a video diffusion model can take sparse input views as anchor frames and "hallucinate" the missing intermediate viewpoints. Because the video model inherently understands how perspective shifts alter a scene, the generated intermediate frames are highly consistent. This converts an ill-posed sparse reconstruction problem into a dense-view optimization problem, which can then be cleanly solved using standard techniques like 3D Gaussian Splatting.
Similarly, Sibo Wu et al. develop GenFusion: Closing the Loop between Reconstruction and Generation via Videos, which uses a video diffusion model to iteratively generate "restoration frames" that repair artifacts and expand boundaries in a reconstructed 3D scene.
Broadening the Temporal-Spatial Pipeline
The success of mapping spatial consistency to temporal video layers has led to several creative extensions across the 3D generation landscape:
Decoupled Pipelines
Instead of optimizing space and time jointly, several frameworks decouple the process for speed. Luke Melas-Kyriazi in IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation and Rishab Parthasarathy in Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion generate a stable 2D video "seed" first, expand each frame into multi-view images, and then reconstruct the 3D assets independently. This achieves high-quality results in a fraction of the time required by joint optimization methods.
Alternative Dimensional Mappings
The mapping does not have to be strictly camera rotation. Tao Hu et al. introduce X-Ray: A Sequential 3D Representation For Generation, which maps depth-ordered surface layers of an object to the temporal dimension of a video file. This allows standard video architectures to predict complex internal and external topologies from a single viewpoint. Conversely, Taegyeong Lee et al. show in Grid Diffusion Models for Text-to-Video Generation that the inverse is also possible: flattening a video's temporal dimension into a spatial grid allows 2D image models to generate consistent video sequences.
Dynamic 4D Supervision
When generating dynamic 4D scenes from text, video models provide a robust global prior. Uriel Singer et al. demonstrate in Text-To-4D Dynamic Scene Generation that supervising a 4D Neural Radiance Field (NeRF) with a video diffusion model ensures that the scene maintains high visual quality and prompt alignment across all viewing angles, preventing the structural degradation common in image-supervised baselines.
By treating the spatial world as a sequence of temporal frames, this research direction elegantly sidesteps the lack of 3D training data, leveraging the vast, rich physics engines implicitly learned by modern video foundation models.
Key Foundational Work
AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatars (2023) — contributed 2 concept elements to this direction.