The central challenge in 3D generative modeling is not just creating a pretty image, but ensuring that the image remains consistent from every possible angle. Traditional 2D diffusion models, while visually stunning, lack an inherent understanding of 3D geometry. When asked to generate multiple views of an object, they often suffer from the "Janus problem"—producing multiple faces on a single head—or "content drift," where the object's identity changes as the camera rotates.
A growing body of research suggests that the solution lies not in more 3D data, which is scarce and expensive, but in Video Diffusion Models (VDMs). By treating 3D generation as a specialized case of video synthesis, researchers are leveraging the temporal consistency of video to enforce spatial consistency in 3D.
The Intuition: Temporal is Spatial
The core insight is that a video of a rotating object is functionally equivalent to a sequence of multi-view images. Because VDMs are trained on massive amounts of real-world footage, they have already learned "commonsense" priors about how light, texture, and geometry behave as a camera moves through space.
Luke Melas-Kyriazi and colleagues in IM-3D argue that these temporal priors are superior to static image priors because they treat multi-view synthesis as a continuous sequence rather than independent frames. This continuity naturally mitigates artifacts like the Janus problem. Similarly, Daniel Watson and the team behind 4DiM found that training on unposed video acts as a powerful spatiotemporal regularizer, allowing models to learn world dynamics and light transport that static datasets cannot provide.
Video Models as Synthetic Data Engines
One of the most successful applications of this idea is using VDMs to bypass the 3D data bottleneck. Instead of training a reconstruction model on limited 3D scans, researchers use a VDM to generate millions of consistent multi-view "videos" of objects.
In VFusion3D, Junlin Han and Philip Torr demonstrate that a Large Reconstruction Model (LRM) trained on 3 million AI-generated multi-view videos can outperform models trained on real 3D data. This approach effectively "distills" the latent 3D knowledge trapped in video models into a structured 3D format. Xiaochuan Li further explores this in Droplet3D, showing that fine-tuning VDMs on 3D data allows them to inherit knowledge of camera motion and object consistency that is missing from smaller, native 3D datasets.
Bridging Generation and Reconstruction
Beyond data generation, video priors are being used to "guide" the optimization of 3D representations like Gaussian Splatting. This is often done through iterative refinement or distillation.
- Iterative Repair: Sibo Wu and Andreas Geiger introduced GenFusion, which uses a VDM to "repair" 3D scenes. By feeding generative "restoration frames" back into a Gaussian Splatting pipeline, the model can fill in holes and resolve artifacts in sparse-view reconstructions.
- Guidance Score Distillation (GSD): In RealisticDreamer, Ruocheng Wu uses VDMs to provide stable, geometrically consistent supervision for 3D reconstruction from as few as three images. By treating novel viewpoints as a continuous video sequence, the model "leashes" the generative process to maintain structural integrity.
- Test-Time Completion: Yan Xu and Stella X. Yu frame novel view synthesis as a "test-time video completion" task. In Novel View Synthesis from A Few Glimpses, they use a VDM to hallucinate intermediate views, which are then used to anchor the 3D geometry.
Scaling to 4D and Large Scenes
The transition from 3D to 4D (3D + time) is a natural extension of this research. Vivid-ZOO by Bing Li factorizes the problem into spatial (3D) and temporal (motion) components, reusing pre-trained layers from existing models to generate dynamic 3D objects from text prompts.
For large-scale environments, models like WonderVerse and FlashWorld leverage video backbones to ensure that as a user "walks" through a generated scene, the environment remains seamless. Xinyang Li shows that this video-first approach can lead to a 10–100× speedup over traditional optimization methods, as the model starts with a much stronger understanding of perspective shifts and object geometry.
By repurposing the "temporal" dimension of video as a "spatial" dimension for 3D, this research direction effectively solves the consistency problem that has long plagued 2D-to-3D generative AI.