TL;DR
ASFM-Net introduces a novel approach to 3D point cloud completion by using an Asymmetrical Siamese Feature Matching strategy. By forcing the network to map incomplete point clouds into the exact same latent space as complete ones, it effectively injects "shape priors" into the feature extraction process. Combined with an iterative refinement unit, the model achieves state-of-the-art performance, including taking 1st place on the Completion3D benchmark and outperforming previous methods by over 12%.
What Problem Does This Paper Address?
In real-world 3D scanning (using LiDAR or depth cameras), the captured point clouds of objects are almost always incomplete and noisy due to self-occlusion or environmental obstructions. Inferring the complete 3D structure from these partial inputs is crucial for downstream tasks like autonomous driving, robotics, and AR/VR.
Most existing learning-based methods (like PCN or TopNet) attempt to solve this by extracting a global feature vector directly and solely from the partial input, and then decoding it into a complete shape. The authors identify a fundamental flaw here: global features extracted only from incomplete inputs are inherently missing geometric details and structural information. If the input is missing half a car, the feature vector simply doesn't know what the other half looks like, leading to blurry or inaccurate reconstructions.
Methods
To solve the missing information problem, the authors propose a coarse-to-fine architecture called ASFM-Net, which consists of two main components:
- Asymmetrical Siamese Auto-encoder: This module learns global "shape priors." It consists of two identical auto-encoders. First, Auto-encoder 1 ($AE_1$) is pre-trained on complete point clouds to map them into a high-dimensional latent space (a codeword), and its weights are then frozen. Next, Auto-encoder 2 ($AE_2$) takes partial point clouds as input. During training, $AE_2$'s encoder is forced to generate a codeword that matches the frozen $AE_1$'s codeword using a Euclidean distance feature matching loss. In inference, the partial point cloud is encoded by $AE_2$, and decoded by $AE_1$'s frozen decoder to generate a coarse, complete shape.
- Iterative Refinement Unit: Because the Siamese auto-encoder generates a coarse output that might lose the fine details of the original input, a refinement unit is introduced. It concatenates the original partial input with the coarse output (using Farthest Point Sampling and XY-plane mirror symmetry) to form a synthetic point cloud. It then uses a 2D grid generator and Multi-Layer Perceptrons (MLPs) to predict coordinate residuals (offsets), shifting the points to reconstruct fine-grained geometric details. This process can be iterated to double the point resolution.
Datasets
The authors trained and evaluated their network on several standard 3D benchmarks:
- PCN Dataset: Derived from ShapeNet, containing 8 categories of paired partial/complete point clouds. Evaluated at a resolution of 4096 points.
- Completion3D Benchmark: A widely used online leaderboard dataset (also derived from ShapeNet) evaluated at a resolution of 2048 points.
- KITTI Dataset: Real-world LiDAR scans of cars used to test the model's ability to generalize from synthetic training data to real, noisy sensor data.
- ShapeNet (Novel Categories): 8 unseen categories (e.g., bed, guitar, skateboard) were used specifically to test the network's zero-shot generalization capabilities.
Key Insights and Ideas
- Latent Space Matching as a "Cheat Code" for Priors: By pre-training an auto-encoder on perfect data and freezing it, the latent space becomes a structured dictionary of valid 3D shapes. Forcing the partial-input encoder to map into this exact same space effectively forces the network to "guess" the closest valid complete shape, rather than trying to hallucinate missing parts from scratch.
- Decoupling Global Shape and Local Details: The Siamese network is great at guessing the overall category and global structure (the prior), but bad at preserving the exact local geometry of the input. The refinement unit acts as a necessary counterbalance, anchoring the hallucinated shape back to the ground-truth partial observations.
- Robustness through Symmetry: The authors found that explicitly incorporating symmetry operations (specifically XY-plane symmetry) when merging the coarse output with the partial input significantly aids the refinement unit in recovering lost details.
Key Results
- Completion3D: ASFM-Net achieved 1st place on the public leaderboard, reaching an average Chamfer Distance (CD-T) of 6.68. This outperformed the second-best method (VRC-Net) by a massive 17.7% margin.
- PCN Dataset: Achieved state-of-the-art results across all 8 categories, improving the average Chamfer Distance (CD-P) by 6.1% compared to the previous best (RFA).
- Extreme Robustness: In ablation studies testing visibility ratios, ASFM-Net successfully reconstructed plausible overall shapes even when only 20% of the original object was visible, a threshold where baseline methods completely failed.
- KITTI Generalization: On real-world LiDAR data, ASFM-Net achieved a fidelity error of 0.025, significantly outperforming TopNet and PCN (0.041), proving it doesn't just memorize synthetic CAD models.
Limitations and Future Work
- Novel Category Misclassification: Because the Siamese auto-encoder relies on learned priors, it can get confused by completely novel objects. For example, the coarse network sometimes mistakenly completes a bed as a chair, or a guitar as a lamp. While the refinement unit usually fixes these errors by pulling the shape back toward the input, the reliance on known-category priors is a limitation for wild, out-of-distribution shapes.
- Future Work: The authors plan to explore how these high-fidelity completed point clouds can be utilized to improve downstream 3D perceptual tasks, such as instance extraction and 3D object type classification.
Summary updated