TL;DR
This paper introduces a novel approach to 3D shape completion that generates multiple, diverse, and plausible full shapes from a single partial point cloud. By leveraging Implicit Maximum Likelihood Estimation (IMLE) instead of traditional GANs, the authors successfully overcome the "mode collapse" problem, producing results that are both structurally varied and geometrically faithful to the input.
What Problem Does This Paper Address?
3D scanning is rarely perfect. Whether due to sensor limitations, occlusions, or sparse data, we often end up with "partial" point clouds—a chair with three legs or a car missing its roof. The goal of shape completion is to fill in these blanks.
Historically, most models treated this as a one-to-one mapping: one partial input equals one "correct" completed output. However, shape completion is inherently ill-posed. If you only see the front of a chair, the back could be rounded, square, or slatted. While Generative Adversarial Networks (GANs) can produce multiple outputs, they often suffer from mode collapse, where the model gets "stuck" and produces nearly identical results regardless of the random noise injected. This paper seeks to generate structurally diverse completions that represent the many ways an object could realistically look.
Methods
The authors propose a conditional IMLE-based framework. Unlike GANs, which use a discriminator to tell "real" from "fake," IMLE ensures that for every real data point in the training set, there is a generated sample nearby. This "coverage-oriented" objective naturally prevents mode collapse.
The Architecture: 1. Autoencoder (E & D): An encoder and decoder are trained to reconstruct 3D shapes using Earth Mover’s Distance (EMD). This creates a meaningful latent space for shapes. 2. Generator (G): This is the core of the multimodal magic. It takes the latent code of a partial shape ($x_i$) and a noise vector ($z_j$) and maps them to a new latent code ($\tilde{y}_{i,j}$). 3. Loss Functions: * $L_1$ Latent Loss: During training, the model generates multiple candidates and pulls the closest one toward the ground-truth latent code. * Unidirectional Hausdorff Distance (UHD): A specific loss used to ensure the generated complete shape physically "encloses" the original partial input, maintaining geometric consistency.
Datasets
The model was evaluated on three primary datasets:
- 3Depn: Derived from ShapeNet using virtual scans to create realistic partial views.
- PartNet: A dataset where entire semantic parts (like a chair back or a table leg) are randomly removed to test the model's ability to "imagine" missing components.
- RobustPointSet: Used to test the model's transferability and its ability to handle noisy, real-world-style artifacts.
Key Insights and Ideas
- IMLE vs. GANs: The most significant insight is that IMLE is a superior alternative to GANs for point cloud generation because it prioritizes "covering" the data distribution. This leads to structural diversity—for a missing chair back, the model might generate a tall back, a short back, or a circular back, rather than just slightly different versions of the same style.
- Implicit Semantics: Even without part-level labels during training, the model learns the "logic" of objects. It understands that if a chair is missing a leg, it needs to add a leg-like structure that matches the existing style.
- Efficiency: Because the model doesn't require a discriminator (which can be notoriously difficult to train), the training and inference times are significantly lower than GAN-based alternatives.
Key Results
The authors compared their method against MSC (a conditional GAN) and kNN-latent baselines.
- Diversity: Using the Total Mutual Difference (TMD) metric, the IMLE approach showed significant gains in diversity. For example, on the 3Depn Chair class, the model achieved a TMD of 2.93 compared to the GAN's 2.56.
- Fidelity: Using Unidirectional Hausdorff Distance (UHD) to measure how well the completion respects the input, the model either outperformed or matched state-of-the-art unimodal and multimodal methods.
- Speed: The model is remarkably fast, requiring only 2 ms per shape for inference, making it suitable for real-time robotics or AR applications.
Limitations and Future Work
The authors acknowledge that their current architecture is relatively simple. While it excels at structural diversity, it struggles to generate very fine-grained, high-resolution details.
Future work could involve:
- Adding refinement networks to sharpen the generated point clouds.
- Applying this IMLE-based approach to other "one-to-many" tasks, such as single-view 3D reconstruction (generating a 3D model from a single 2D image).
- Exploring how to add fine-grained styles and textures in a multimodal fashion.
Summary updated