Geometry Aware Diffusion for Consistent View Synthesis

This direction is merged into another direction: Janus problem. Its last version stays readable and citable here.

Version 1 of 1219 papers228 conceptsfrozen 2026-10-04

If you ask a modern image generator to create a picture of a cat, it succeeds brilliantly. If you ask it to generate ten pictures of the same cat from ten different angles, it almost certainly fails. The ears might change shape, the fur pattern might shift, or, most infamously, the cat might end up with two faces—a phenomenon researchers call the Janus problem.

This lack of "multi-view consistency" is the primary hurdle in transforming powerful 2D diffusion models into 3D content creators. Recent research has converged on a shared solution: anchoring the "imagination" of 2D models to the rigid laws of 3D geometry.

The Problem of Independent Sampling

Standard diffusion models like Stable Diffusion are inherently 2D. They generate pixels based on local statistical patterns rather than a global understanding of volume. When these models are used to synthesize novel views, they often treat each camera angle as an independent creative task.

As noted by Jianglong Ye in Consistent-1-to-3, this independence leads to "flicker" and shape-shifting. Because the model lacks a shared 3D structure, a feature generated in one view (like a teapot handle) might mysteriously migrate or disappear in the next. Yuan Liu and colleagues demonstrate in SyncDreamer that without synchronization, the generative process produces "physically contradictory" images that cannot be fused into a coherent 3D object.

Anchoring to Structural Priors

To fix this, researchers are moving away from "pure" generation and toward geometry-aware diffusion. The intuition is simple: give the diffusion model a "skeleton" to wrap its textures around.

One effective strategy is to use explicit 3D models as anchors. For human generation, Zhenzhen Weng introduced ZeroAvatar, which ties the diffusion process to a SMPL body model. By grounding the generative "hallucinations" in a predefined human shape, the model avoids generating multiple faces. Similarly, Yushi Lan in Gaussian3Diff maps 3D Gaussians onto a parametric face model, ensuring that expressions and features remain consistent across all viewing angles.

Enforcing Consistency via Epipolar Geometry

When explicit models aren't available, researchers use the mathematical constraints of stereo vision—epipolar geometry. If you know the camera positions for two views, a point in the first image must lie along a specific line (the epipolar line) in the second.

Zi-Xin Zou and the team behind Sparse3D use these epipolar features to guide the denoising process, forcing the model to maintain pixel-level alignment. Zehuan Huang takes this further in EpiDiff by inserting specialized attention blocks into the diffusion network. These blocks allow the model to "look" at neighboring views along epipolar lines, ensuring that a button on a shirt stays in the same 3D location across the entire generated sequence.

Closing the Loop with Recursive Diffusion

A final, powerful intuition is the "generate-reconstruct" feedback loop. In traditional pipelines, you generate images and then try to build a 3D model from them. If the images are inconsistent, the 3D model fails.

Hao Wen and Yaohui Wang proposed Ouroboros3D, which treats 3D generation as a recursive process. The intermediate 3D reconstruction is fed back into the diffusion model as a geometric condition for the next step of denoising. This "recursive" approach bridges the domain gap, gradually forcing the 2D images to conform to a single, valid 3D geometry.

By moving from unconstrained 2D generation to geometry-conditioned synthesis, these methods transform diffusion models from mere "image makers" into engines for consistent, high-fidelity 3D worlds.

Papers in this version

Showing 60 of 219 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_geometry_aware_diffusion_for_consistent__4d947705,
  title = {Geometry Aware Diffusion for Consistent View Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometry-aware-diffusion-for-consistent-view-synthesis/rdr_4d947705a21e4ae98ba0577a7a1ff31d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_geometry_aware_diffusion_for_consistent__4d947705_v1,
  title = {Geometry Aware Diffusion for Consistent View Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometry-aware-diffusion-for-consistent-view-synthesis/rdr_4d947705a21e4ae98ba0577a7a1ff31d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}