When we look at a 3D object from a single perspective, or when a LiDAR sensor scans a street, we only see the "front" surfaces. The back and internal structures are hidden—occluded by the object itself or other items in the scene. Reconstructing the full 3D geometry from these partial observations is a fundamental challenge in computer vision.
The core difficulty is that this problem is inherently ill-posed. A single partial scan does not map to one unique complete shape; instead, it could correspond to many different, equally valid geometries. This is known as a "one-to-many" mapping problem.
The Failure of Deterministic Regression
Traditional deep learning approaches to 3D reconstruction often treat the task as a standard regression problem. They use an encoder to compress the partial input and a decoder to predict a single "ground truth" shape. However, as Haoqiang Fan, Hao Su, and Leonidas Guibas observed in their seminal work on Point Set Generation Networks, deterministic models struggle with uncertainty.
When a model is trained to minimize the distance (like Mean Squared Error or Chamfer Distance) between its prediction and a single ground truth, but the input is ambiguous, the model tends to hedge its bets. It produces a blurry, "average" shape that attempts to be close to all possible outcomes at once. This result often fails to represent any single realistic geometry, leading to reconstructions that lack sharp features or physical plausibility.
Capturing the Distribution of Plausible Shapes
To solve the "average shape" problem, researchers have shifted toward generative modeling. Instead of predicting one shape, these models learn the underlying probability distribution of all complete shapes that could fit a partial observation.
Early efforts in this direction utilized Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs). For instance, Or Litany and Michael Bronstein used Graph Convolutional Autoencoders to learn a latent space of realistic human body shapes. By performing optimization within this learned "prior" space, they could find multiple completions that were both realistic and consistent with the partial scan.
Similarly, Rundi Wu and colleagues introduced Multimodal Shape Completion via cGANs. By conditioning a GAN on the partial input and a latent "style" code, they enabled the network to "hallucinate" diverse, high-quality completions, effectively moving from "one-to-one" reconstruction to "one-to-many" generation.
Diffusion Models and High-Fidelity Hallucination
The most recent leap in this field comes from Denoising Diffusion Probabilistic Models (DDPMs). Diffusion models are particularly well-suited for 3D completion because they excel at capturing complex, multi-modal distributions without the training instabilities of GANs.
In Point-Voxel Diffusion (PVD), Linqi Zhou and Jiajun Wu demonstrated that by treating 3D points as a distribution to be denoised, a model can generate highly detailed and diverse completions. This approach was further refined by Christian Rupprecht and Andrea Vedaldi in PC², which projects local image features onto the evolving 3D points during the diffusion process. This ensures that the "hallucinated" parts of the object remain geometrically consistent with the visible parts.
Leveraging Priors and Scene Constraints
A key insight across this research is that the model shouldn't just guess blindly; it should use priors—pre-existing knowledge of what objects look like.
- GAN Inversion: Junzhe Zhang and Ziwei Liu developed ShapeInversion, which uses a GAN pre-trained on complete objects. To complete a partial scan, the system searches for the latent code in the pre-trained GAN that best matches the visible parts, effectively "filling in" the rest using the GAN's internal knowledge of object structure.
- Self-Supervision: To bridge the gap between clean synthetic data and messy real-world scans, RealDiff by Başak Melis Öcal uses self-supervised diffusion. It leverages multi-view consistency and depth priors to learn how to complete shapes directly from noisy sensor data.
- Scene Context: Wesley Khademi has shown that completion shouldn't happen in a vacuum. By incorporating scene constraints—such as knowing where "free space" is (where the object cannot be)—models can generate completions that are not only plausible in isolation but also physically valid within their environment.
By moving from deterministic "best guesses" to probabilistic distributions, these researchers have transformed 3D reconstruction from a rigid geometric task into a flexible process of informed imagination.