When reconstructing a 3D scene or object from limited visual data—such as a single image, a sparse set of viewpoints, or a monocular video—computer vision models inevitably run into the problem of occlusion. Traditional 3D reconstruction techniques like Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS) rely heavily on multi-view consistency. They operate by interpolating information across overlapping camera views.
However, when a region is entirely unobserved (e.g., the back of a person captured from a single frontal camera, or an object hidden behind a hand), the mathematical optimization problem becomes severely under-constrained. As noted by Georg Hess et al. and Shenglun Chen et al., standard geometric pipelines lack the capacity to synthesize large-scale missing structures without external knowledge, often leaving behind hollow, messy, or unrealistically blurred surfaces.
To solve this, a rapidly growing body of research integrates generative diffusion priors into 3D reconstruction loops. Instead of treating reconstruction purely as a geometric matching problem, these methods treat it as a task of grounded hallucination. By leveraging the massive statistical distributions learned by 2D and 3D diffusion models, these frameworks can infer physically and semantically plausible geometry for regions they cannot see.
Distilling 2D Generative Priors into 3D Space
One of the most prominent strategies for guessing unseen geometry is to use a pre-trained 2D text-to-image or view-conditioned diffusion model as a "multi-view supervisor." Rather than generating a static 2D image, the diffusion model acts as a loss function that guides the optimization of an underlying 3D representation.
[Partial 2D Views] ──> [3D Representation (Splatting/Implicit)] ──> [Render Unobserved View]
│
▼
[Plausible 3D Surface] <── [Update Gradients] <── [2D Diffusion Prior (SDS/Loss)]
To bridge the gap between 2D image pixels and 3D structural surfaces, researchers often distill geometric signals rather than raw RGB colors alone. For example, Savva Ignatyev et al. introduce a multi-view Score Distillation Sampling (SDS) approach in NeuSD. By rendering a grid of normal maps from both observed and unobserved viewpoints and passing them through a diffusion model, the system generates gradients that push the neural implicit surface toward a plausible, watertight completion without requiring any fine-tuning of the underlying diffusion model.
When applied to dynamic scenes, this multi-view supervision allows models to hallucinate moving components that pass out of frame. Hengyi Wang et al. and Inhee Lee et al. combine the rapid rendering capabilities of 3D Gaussian Splatting with view-conditioned diffusion models. When a monocular video tracks a moving subject, the diffusion prior supervises the unobserved angles, hallucinating realistic textures and geometry for hidden areas (like a person's back) while maintaining strict metric fidelity in the visible regions. To ensure that these hallucinated geometries remain anatomically plausible during complex, non-rigid movements, Xinzhou Wang et al. introduce Canonical Score Distillation, which anchors the generative updates to a consistent canonical 3D skeleton.
Hallucination Before Reconstruction: 2D Inpainting Pipelines
A alternative intuition is to fix the visual data in the 2D domain before attempting to lift it into 3D space. This is often framed as a "reconstruction-before-hallucination" or "canvas-extension" pipeline, ensuring that the subsequent 3D optimization is grounded in a complete visual scaffold.
As demonstrated by Yiming Li et al., identifying visible structures first provides a reliable foundation that prevents the generative model from hallucinating content that contradicts the ground truth. Following this philosophy, Junsheng Zhou et al. deploy generative diffusion models to explicitly inpaint the missing parts of segmented objects within a single-view image. Once the 2D image is rendered whole, standard 3D mesh generation tools can reconstruct the complete geometry without being tripped up by the original occlusions.
[Occluded 2D Image] ──> [2D Generative Inpainting] ──> [Complete 2D Canvas] ──> [3D Mesh Lifting]
This pipeline is particularly powerful for resolving severe, localized occlusions, such as those found in human-object interactions. Bin Wen et al. utilize 2D generative inpainting to hallucinate parts of an object that are completely obscured by a user's hands, allowing 3D reconstruction tools to resolve the full spatial relationship and contact points of the interaction.
On a broader scale, this concept applies to scene expansion and novel view synthesis. Guo Pu et al. use a 2D generative model to extrapolate scene content far beyond the boundaries of a single input image. This expanded canvas serves as a consistent visual scaffold for building an expanded Multiplane Image (MPI), allowing for drastically wider camera movements. Similarly, Luxi Chen trains video-to-video diffusion models on incomplete scene renderings, teaching the model to progressively expand a persistent 3D Gaussian Splatting representation during full 360-degree camera rotations.
Direct 3D Diffusion and Structural Scaffolding
Instead of relying on 2D projections and image-space losses, some frameworks apply the denoising diffusion process directly within the 3D domain. This approach bypasses the perspective ambiguities of 2D-to-3D lifting by learning the native spatial distributions of point clouds, voxels, or boundary representations.
The core intuition behind direct 3D shape completion is to treat the observed geometric data as an immutable anchor. Linqi Zhou et al. pioneered this with Point-Voxel Diffusion (PVD). During the conditional denoising cycle, the points corresponding to the observed, visible geometry are kept completely fixed. The diffusion model applies its denoising steps exclusively to the missing regions, allowing it to sample multiple diverse, structurally valid completions that seamlessly snap onto the true boundaries of the partial input.
Initial State: [ Fixed Observed Points ] + [ Pure Random Noise Region ]
│
▼ (3D Denoising Steps)
Final Surface: [ Fixed Observed Points ] + [ Hallucinated Coherent Structure ]
This direct 3D scaffolding scales from individual objects to massive environments. For instance, Lucas Nunes formulates the 3D diffusion process as a series of local, point-wise offsets tailored for sparse LiDAR scene completion in autonomous driving. By treating the existing sparse sensor points as a structural scaffold, the model fills in massive environmental gaps locally, bypassing the memory constraints of global voxel grids.
When the missing geometry is highly uncertain, a single deterministic guess can lead to physically impossible structures. To counter this, Donghwan Kim et al. introduce a multi-hypothesis point cloud diffusion model for human reconstruction. The diffusion process generates multiple distinct structural proposals for occluded limbs, ensuring that whatever component is synthesized aligns globally with the body's overall geometric distribution rather than relying on a single, potentially flawed implicit function. Even in highly structured engineering domains, Xiang Xu demonstrates that diffusion models operating on hierarchical tree structures can perform automated CAD boundary representation (B-rep) completion, morphing between distinct topological shapes while preserving valid geometric properties.