Pre-trained 2D diffusion models are remarkably good at "imagining" textures, lighting, and semantic details. However, as Paul Yoo and colleagues observe in DreamSparse, these models are like the prisoners in Plato’s Allegory of the Cave: they see only 2D shadows of a 3D reality. Because they lack an inherent understanding of 3D geometry, they often produce "hallucinations" when asked to maintain consistency across different viewpoints.
The research direction emerging from these papers focuses on "lifting" these powerful 2D priors into the 3D world, ensuring that generated images obey the laws of perspective, occlusion, and spatial logic.
The Geometry Gap and the Janus Problem
The fundamental challenge is that a standard 2D diffusion model treats every viewpoint as an independent generation task. Without a shared spatial memory, the model might generate a visually stunning front view of a person, but when asked for a side view, it might generate another front view—a phenomenon known as the "Janus problem" or multi-faced artifact.
As Jinbo Wu notes in HD-Fusion, this lack of spatial awareness makes it difficult to maintain global structural coherence. Similarly, Thu Nguyen-Phuoc and Long Mai argue in BlockGAN that 2D-only models struggle with complex spatial phenomena like foreshortening and perspective-correct shadows.
Structural Scaffolding: Injecting 3D Inductive Biases
To bridge this gap, researchers are moving away from pure 2D generation toward architectures that incorporate 3D "scaffolding." This involves using a coarse 3D representation to guide the 2D diffusion process.
One approach is to use a 3D structural prior as a skeleton. In 3DFuse, Junyoung Seo and Jiyoung Lee inject sparse point clouds or depth maps into the diffusion process. This "skeleton" ensures that the model knows where the object is in space before it starts painting the pixels. Similarly, ControlRoom3D by Jonas Schult uses 3D "proxy rooms" (bounding boxes) to prevent the model from hallucinating multiple beds or floating furniture in a generated scene.
Another method involves Joint Distribution Modeling. JointNet, developed by Jingyang Zhang and Shiwei Li 0001, trains a parallel branch to learn images and dense labels (like depth or surface normals) simultaneously. By "locking" the original image branch, they preserve the model's creativity while forcing it to be 3D-aware.
Synchronized Attention: Forcing Multi-View Consistency
If the problem is that views are generated independently, the solution is to make them "talk" to each other during the generation process.
SyncDreamer, by Yuan Liu and Cheng Lin, introduces a 3D-aware attention mechanism that synchronizes the denoising process across all views at once. Instead of generating one view and then another, the model generates a "circle" of views that are mathematically constrained to be consistent.
SPAD (Spatially Aware Multi-View Diffusers) takes this further by using Epipolar Attention. Yash Kant and Ziyi Wu use the principles of geometry—specifically how points in one camera view must correspond to lines in another—to "upcycle" 2D models into 3D-consistent generators. This prevents the "content copying" where the same feature appears in the wrong place across different angles.
Lifting and Distillation: From Pixels to Volumes
A final major theme is the use of Score Distillation Sampling (SDS) and its variants to transfer 2D knowledge into explicit 3D representations like Neural Radiance Fields (NeRF) or 3D Gaussians.
In NeuralLift-360, Dejia Xu and Yifan Jiang use a 2D diffusion model to "hallucinate" the hidden sides of a 3D object, while a NeRF backbone ensures the result is structurally sound. InFusion by Nan Xue applies this to 3D Gaussian Splatting, using diffusion priors to fill in missing parts of a 3D scene with 20x the speed of previous methods.
By combining the "imagination" of 2D models with the "rigidity" of 3D representations, these researchers are creating systems that can not only generate beautiful images but also understand the three-dimensional world they represent.