Abstract
Shape completion is the problem of completing partial input shapes such as partial scans. This problem finds important applications in computer vision and robotics due to issues such as occlusion or sparsity in real-world data. However, most of the existing research related to shape completion has been focused on completing shapes by learning a one-to-one mapping which limits the diversity and creativity of the produced results. We propose a novel multimodal shape completion technique that is effectively able to learn a one-to-many mapping and generates diverse complete shapes. Our approach is based on the conditional Implicit Maximum Likelihood Estimation (IMLE) technique wherein we condition our inputs on partial 3D point clouds. We extensively evaluate our approach by comparing it to various baselines both quantitatively and qualitatively. We show that our method is superior to alternatives in terms of completeness and diversity of shapes.
Introduction
3D scanning in real-world environments is rarely perfect. Sensor limitations, occlusions, and data sparsity often result in "partial" point clouds—for instance, a chair missing its back or a car missing its roof. The task of shape completion aims to reconstruct the full geometry from these incomplete observations.
Historically, shape completion has been treated as a deterministic one-to-one mapping. However, the problem is inherently ill-posed: a single partial view can correspond to many plausible full shapes. While Generative Adversarial Networks (GANs) can theoretically produce multiple outputs, they frequently suffer from "mode collapse," where the generator produces nearly identical results regardless of the input noise. This research introduces a framework to generate structurally diverse completions that realistically represent the potential variations of an object.
Methodology
The proposed framework leverages conditional Implicit Maximum Likelihood Estimation (IMLE). Unlike GANs, which rely on a discriminator, IMLE uses a coverage-oriented objective. It ensures that for every real data point in the training set, there is a generated sample nearby, which naturally prevents mode collapse and encourages diversity.
Architecture
- Autoencoder (E & D): An encoder and decoder are pre-trained to reconstruct 3D shapes using Earth Mover’s Distance (EMD), establishing a robust latent space.
- Generator (G): The core component takes the latent code of a partial shape ($x_i$) and a random noise vector ($z_j$) to map them to a new latent code ($\tilde{y}_{i,j}$) representing a complete shape.
- Loss Functions: The model employs an $L_1$ Latent Loss to pull generated candidates toward ground-truth codes. Additionally, a Unidirectional Hausdorff Distance (UHD) loss ensures the generated shape physically encloses the original partial input, maintaining geometric consistency.
Experimental Setup
The model was evaluated across three diverse datasets:
- 3Depn: Created from ShapeNet using virtual scans for realistic partial views.
- PartNet: Focused on semantic part removal (e.g., removing a chair back) to test structural "imagination."
- RobustPointSet: Used to evaluate performance on noisy, real-world-style data.
Key Results
The IMLE-based approach was compared against multimodal GAN baselines (MSC) and kNN-latent methods.
- Structural Diversity: Using the Total Mutual Difference (TMD) metric, the IMLE approach showed significant improvements. On the 3Depn Chair class, it achieved a TMD of 2.93, outperforming the GAN's 2.56.
- Geometric Fidelity: The model matched or exceeded state-of-the-art methods in Unidirectional Hausdorff Distance (UHD), indicating high respect for the input geometry.
- Inference Speed: The model is highly efficient, with an inference time of approximately 2 ms per shape, making it viable for real-time applications in robotics and augmented reality.
Conclusion and Future Work
By prioritizing data distribution coverage over adversarial competition, this IMLE-based method successfully generates diverse and faithful 3D shape completions. While the current architecture excels at structural variety, future work will focus on improving high-resolution detail through refinement networks and extending the approach to single-view 3D reconstruction from 2D images.
Summary updated