Reconstructing a complete 3D scene from a single 2D image is a fundamentally "ill-posed" problem. As Yuxiao Li notes in A Modular Framework for Single-View 3D Reconstruction of Indoor Environments, a single viewpoint cannot capture what lies behind a sofa or inside a cabinet. Traditional reconstruction methods often produce "hollow shells"—geometry that looks correct from the starting camera angle but dissolves into a digital void as soon as the perspective shifts.
A emerging research direction solves this by treating 3D reconstruction not just as a geometric estimation task, but as a generative "hallucination" task. By orchestrating 2D diffusion models to perform amodal completion—the ability to perceive the whole of an object even when only a portion is visible—researchers are bridging the gap between partial 2D observations and watertight 3D assets.
The Inpaint-then-Lift Pipeline
The core intuition behind this direction is that 2D generative models, trained on billions of images, have developed a form of "spatial common sense." They know what the back of a chair or the hidden side of a cup should look like.
Instead of trying to project a partial, occluded image directly into 3D, frameworks like Deep Prior Assembly by Junsheng Zhou et al. use a "committee" of specialized models. They first segment the visible parts, then use generative inpainting to hallucinate the occluded regions in 2D. This creates a "complete" visual prior. Only after the 2D image is "healed" is it "lifted" into 3D geometry.
This sequence is critical. Yukai Shi and colleagues demonstrate in SceneMaker that generating missing parts in 2D before reconstruction produces significantly higher quality geometry than attempting to generate 3D structures directly from occluded inputs. The 2D de-occlusion stage provides a robust foundation that prevents the geometric distortions typically caused by incomplete visual data.
Resolving Human-Object Interactions
One of the most challenging environments for 3D reconstruction is the "in-the-wild" interaction between humans and objects. In these scenes, hands and bodies frequently block large portions of the objects being manipulated.
To solve this, EasyHOI, developed by Xiaoxiao Long and Yuan Liu, utilizes a diffusion-based inpainting model to "remove" the interacting hand and hallucinate the missing geometry of the object. Similarly, Open3DHOI by Bin Wen et al. integrates 2D inpainting into a 3D Gaussian Splatting pipeline. By restoring the full appearance of an object in image space, these models allow 3D optimizers to work with a complete visual representation, ensuring that the resulting 3D meshes are physically plausible and contact-accurate.
From Static Shells to Interactable Assets
The value of hallucinating occluded regions extends beyond aesthetics; it is a requirement for interactivity. In DreamArt, Ruijie Lu and Yu Liu show that for articulated objects—like a laptop that opens or a drawer that slides—the system must pre-generate the geometry and textures of surfaces hidden behind the movable parts. Without this amodal completion, opening a digital drawer would reveal a blurry, stretched texture artifact instead of a realistic interior.
This "hallucination" also powers robotic perception. G3Flow uses 3D generative models to maintain a "digital twin" of an object. By completing the geometry of hidden functional parts (like a tool handle), the robot can maintain a consistent semantic understanding of the object even when its own gripper occludes the view.
Believing is Seeing
Perhaps the most ambitious extension of this idea is unobserved object detection. In Believing is Seeing, Dylan Campbell et al. argue that generative models can predict the location of objects that are not just occluded, but entirely outside the camera's field of view.
By leveraging the spatial priors of diffusion models, the system can infer that a "living room" scene likely contains a hidden wall or a nearby piece of furniture just off-screen. This suggests that the future of 3D reconstruction lies in moving away from purely "seeing" what is there, toward "reasoning" about what must be there to make the scene complete.