When a computer vision system or a robot observes the physical world, it rarely sees complete objects. Due to perspective limits, sensor sparsity, and foreground obstructions, the incoming data is almost always a partial snapshot—a 2.5D depth map, a sparse LiDAR scan, or a single-view RGB image.
Inferring the complete 3D structure from these partial observations is known as shape completion or amodal perception. Historically, models treated this as a deterministic regression task: given a partial input, predict the single "correct" complete shape. However, this framing fundamentally misunderstands the physics of occlusion. Shape completion is inherently ill-posed and ambiguous; a single partial view can correspond to many distinct, equally valid full geometries.
To resolve this, a major research direction has shifted away from deterministic reconstruction toward probabilistic shape completion. By treating completion as a one-to-many generative problem, these methods model the entire distribution of plausible shapes, avoiding the structural failures of traditional approaches.
The Flaw of the Deterministic Average
To understand why probabilistic models are necessary, we must first look at why deterministic models fail.
When a deterministic network is trained to complete a shape, it typically minimizes a distance metric—such as Mean Squared Error (MSE), Chamfer Distance, or Binary Cross-Entropy—between its prediction and the ground truth. If the training data contains multiple valid ways to complete a partial shape (for instance, a coffee mug whose hidden handle could be on the left, the right, or absent entirely), a deterministic model faces a mathematical trap. To minimize the global loss across all these possibilities, the network is forced to output the mathematical mean of all valid completions.
As Christian Rupprecht et al. observe in their work on single-image reconstruction, PC2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction, this averaging strategy results in a "single average guess" that is often physically implausible. Instead of a sharp, realistic object, the model generates a blurry, washed-out shape that lacks high-frequency details.
Or Litany et al. emphasize this in Deformable Shape Completion With Graph Convolutional Autoencoders, noting that because a single partial scan corresponds to multiple plausible complete shapes, standard deterministic models cannot effectively navigate this geometric ambiguity. The result is an inaccurate or overly smoothed reconstruction that fails to represent any real-world object.
Deterministic Approach:
Partial Input ───> [ Deterministic Model ] ───> Blurry Average Shape (Mean of all possibilities)
Probabilistic Approach:
┌───> Sample 1 ───> Plausible Shape A (e.g., Straight-back chair)
Partial Input ───> │───> Sample 2 ───> Plausible Shape B (e.g., Curved-back chair)
└───> Sample 3 ───> Plausible Shape C (e.g., Stool)
Mapping One-to-Many Distributions
Probabilistic shape completion redefines the objective. Instead of learning a rigid function $f(\text{partial}) = \text{complete}$, the goal is to model the conditional probability distribution $p(\text{complete} \mid \text{partial})$.
By framing completion as a one-to-many mapping, the system can use stochastic sampling to explore the full range of valid structures that could exist behind a visible surface. As Rundi Wu et al. demonstrate in Multimodal Shape Completion via Conditional Generative Adversarial Networks, this approach allows a system to generate multiple diverse, high-quality completions for a single partial 3D scan, capturing the natural uncertainty of occluded data rather than forcing a single blurred compromise.
This preservation of detail is a direct consequence of the generative loss functions. Because generative models (like GANs, VAEs, or Diffusion Models) evaluate whether a sample looks real and matches the distribution, they penalize non-physical averages. As Dongsu Zhang et al. explain in Probabilistic Implicit Scene Completion, probabilistic modeling maintains high-frequency details even in regions with significant missing data because it forces the model to commit to distinct, sharp geometric outcomes rather than blending them together.
Structuring the Latent Space
To generate these diverse shapes effectively, researchers have devised several ways to structure and sample from the underlying geometric distributions.
High-Dimensional Latent Embeddings
One strategy is to map partial shapes into a continuous, well-behaved latent space where variations correspond to valid geometric deformations. For example, Chenyangguang Zhang et al. introduce a method in U-RED: Unsupervised 3D Shape Retrieval and Deformation for Partial Point Clouds that maps potential shape completions onto a high-dimensional unit sphere. By sampling from this sphere, the system can propose various full shapes that match the partial input, transforming a rigid retrieval task into a flexible probabilistic selection process.
Normalizing Flows and Energy-Based Models
Other architectures focus on explicitly modeling the density of the shape space. In Diverse Plausible Shape Completions from Ambiguous Depth Images, Brad Saund integrates normalizing flows into a Variational Autoencoder (VAE). This allows a robot to output a discrete set of diverse, plausible candidates from a single depth image, ensuring that downstream controllers can plan for multiple valid interpretations of an obstacle.
Alternatively, Ruikai Cui et al. utilize an Energy-Based Model (EBM) within the latent space in Energy-Based Residual Latent Transport for Unsupervised Point Cloud Completion. By using a residual sampling strategy, their model maps out explicit "uncertainty maps" that show exactly which regions of the reconstructed shape are highly confident and which are highly ambiguous.
The Power of Diffusion Priors
Recently, Denoising Diffusion Probabilistic Models (DDPMs) have become a dominant mechanism for probabilistic shape completion. Diffusion models excel here because they break the generation process down into hundreds of small, iterative denoising steps, allowing the model to gradually "imagine" missing details while constantly anchoring the generation to the observed features.
In 3D Shape Generation and Completion through Point-Voxel Diffusion, Linqi Zhou et al. introduced Point-Voxel Diffusion (PVD). By combining a hybrid point-voxel representation with a diffusion process, PVD handles the one-to-many nature of 3D reconstruction by synthesizing entirely distinct, valid shapes from the same partial starting point.
To make these systems practical for real-world robotics, Katherine Liu et al. developed OmniShape, presented in OmniShape: Zero-Shot Multi-Hypothesis Shape and Pose Estimation in the Real World. OmniShape uses a two-stage diffusion process to generate multiple distinct geometric hypotheses. The system then evaluates these hypotheses against the sensor's raw depth data, selecting the candidate that demonstrates the highest geometric alignment.
While diffusion models provide exceptional fidelity, their iterative nature can be computationally slow. To address this, Wesley Khademi et al. proposed a style-modulated conditional GAN in Diverse Shape Completion via Style Modulated Generative Adversarial Networks that uses multi-scale diversity penalties to capture the same geometric uncertainty up to 500 times faster than diffusion alternatives.
Scaling to Scenes and Segmentation
The shift from deterministic to probabilistic modeling extends beyond individual objects to full-scene understanding and 2D image analysis.
- Semantic Scene Completion: In autonomous driving, vehicles must infer full 3D environments from sparse, occluded sensor data. Helin Cao et al. apply diffusion models to this problem in DiffSSC: Semantic LiDAR Scan Completion using Denoising Diffusion Probabilistic Models, showing that modeling the distribution of unobserved regions prevents the unrealistic, broken scene representations common to older, deterministic LiDAR models.
- Amodal Segmentation: In the 2D domain, amodal segmentation requires a model to predict the full mask of an object, including the parts hidden behind foreground occlusions. Early on, Ke Li et al. noted in Amodal Instance Segmentation that deterministic masks fail to account for the inherent ambiguity of articulated parts, which could exist in multiple valid poses behind a barrier. More recently, Kaihua Chen tackled this in Using Diffusion Priors for Video Amodal Segmentation by repurposing pre-trained video diffusion models as structural priors, allowing the system to generate diverse, temporally consistent completions of hidden regions across video frames.
By shifting the problem framing from prediction to distribution modeling, probabilistic shape completion provides a mathematically rigorous and physically realistic solution to the challenge of geometric occlusion. Instead of guessing a single, potentially incorrect average, modern vision systems can now map out, sample, and reason about the entire spectrum of what might be hidden from view.
Key Foundational Work
Amodal Instance Segmentation (2016) — contributed 1 concept elements to this direction.
A Bayesian Approach to Perceptual 3D Object-Part Decomposition Using Skeleton-Based Representations (2015) — contributed 1 concept elements to this direction.