Generative Priors for Sparse-View 3D Reconstruction

This direction is merged into another direction: generative geometric priors. Its last version stays readable and citable here.

Version 1 of 158 papers67 conceptsfrozen 2026-10-04

Traditional 3D reconstruction is a game of triangulation. To build a digital model of an object, algorithms like Structure-from-Motion or Neural Radiance Fields (NeRF) typically require dozens or hundreds of images from different angles. They look for matching pixels across views to calculate depth and geometry.

But what happens when you only have one photo? Or a short video where the camera never sees the back of the subject? In these "sparse-view" or "monocular" scenarios, the geometric problem is mathematically underspecified—there are infinitely many 3D shapes that could explain a single 2D image.

Researchers are solving this by shifting the problem from pure geometry to generative hallucination. By using large-scale generative models (like Stable Diffusion) as "priors," AI can now "imagine" the unobserved parts of a scene—the back of a head, the interior of a cabinet, or the far side of a street—with startling realism.

The Modality Gap and the Need for Priors

In a survey of the field, Fangneng Zhan and colleagues describe this as bridging the "modality gap." When visual data is missing, the system must rely on "spatial common sense" learned from millions of other images.

For example, if you see a person from the front, you know they likely have hair on the back of their head. A traditional algorithm sees a "black hole" of data; a generative model sees a prompt. As Jack Saunders notes in GASP, using a prior trained on 360-degree data allows a model to predict consistent features across the entire geometry, even from a single viewpoint.

Score Distillation: The Bridge Between 2D and 3D

The most influential mechanism for linking 2D generative power to 3D space is Score Distillation Sampling (SDS).

Imagine you are sculpting a clay bust of a person you’ve only seen from the front. You have a "critic" standing behind you who has seen millions of people. As you sculpt the back of the head, the critic doesn't tell you exactly where to put the clay; instead, they say, "That doesn't look like a real head, try making it more rounded."

In this analogy:

  1. The 3D Representation (often 3D Gaussian Splatting) is the clay.
  2. The Generative Model (like a 2D Diffusion model) is the critic.
  3. SDS is the feedback loop.

Dejia Xu and the team behind NeuralLift-360 demonstrated that by distilling the knowledge of a 2D diffusion model into a 3D volume, you can "lift" a single photo into a complete 360-degree object. The diffusion model provides a "supervisory signal" for viewpoints the camera never actually visited.

From Static Objects to Dynamic 4D Scenes

The frontier of this research is moving from static objects to 4D reconstruction—capturing 3D geometry that moves over time from a single video.

In Consistent4D, Yanqin Jiang frames 4D reconstruction as a generative task. Instead of trying to solve the motion through complex physics alone, the system uses 3D-aware diffusion models to hallucinate what a moving object looks like from the side while it's performing an action.

Similarly, Wen-Hsuan Chu and Lei Ke introduced GenMOJO, which reconstructs interacting objects even when they are heavily occluded. By centering individual objects and applying object-centric diffusion priors, the system can "fill in" a dog's legs even if they are hidden behind a couch in the original video.

Applications in Robotics and Simulation

This ability to "see the unseen" is transformative for autonomous systems.

  • Robot Navigation: Weiming Zhi and Ziyong Ma developed VGER, which uses video generation to "imagine" a fly-through of a room from a single photo. This allows a robot to plan a path around a table it hasn't fully seen yet.
  • Unobserved Object Detection: Dylan Campbell explores Believing is Seeing, where generative models use "spatial common sense" to predict the location of objects entirely outside the camera's field of view.
  • Digital Twins: For complex materials like fluids, FluidNexus uses video diffusion to synthesize missing perspectives, allowing for the reconstruction of 3D splashes and flows from a single-view camera.

The Shift in Perspective

The common thread across these 56 papers is a fundamental shift in how we think about 3D vision. We are moving away from reconstruction by measurement (relying only on what is visible) toward reconstruction by synthesis (using learned priors to complete the picture).

As Qitao Zhao and Shubham Tulsiani argue in SparseAGS, this generative approach prevents 3D models from "collapsing" into flat shapes when data is sparse. By trusting the "imagination" of the model as much as the pixels in the image, we can finally build complete digital worlds from the most minimal of glimpses.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generative_priors_for_sparse_view_3d_rec_8c523320,
  title = {Generative Priors for Sparse-View 3D Reconstruction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-priors-for-sparse-view-3d-reconstruction/rdr_8c523320c60c49e6bc381c476ab4ebb7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_generative_priors_for_sparse_view_3d_rec_8c523320_v1,
  title = {Generative Priors for Sparse-View 3D Reconstruction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-priors-for-sparse-view-3d-reconstruction/rdr_8c523320c60c49e6bc381c476ab4ebb7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}