Abstract
ComplementMe is a neural framework designed to automate 3D shape assembly by suggesting both the next component to add and its precise placement. Unlike previous methods that require expensive, manually labeled datasets, ComplementMe learns from unlabeled online repositories using a weakly-supervised approach. By utilizing a probabilistic retrieval model, the system effectively handles the inherent ambiguity of design, offering multiple creative paths for 3D modeling.
Introduction
Creating 3D models from scratch is a high-barrier task requiring significant expertise. "Assembly-based modeling"—where users build shapes by kit-bashing parts from existing libraries—is a popular alternative for novices. However, as libraries grow into the thousands, finding the perfect matching part and aligning it manually becomes a "needle-in-a-haystack" problem.
Existing suggestion systems typically rely on heavy supervision, requiring datasets where every part is perfectly segmented and semantically labeled. Creating these datasets is slow and expensive. Furthermore, 3D modeling is non-functional; there isn't just one "correct" part for a partial assembly. A standard neural network trying to predict a single "average" part would likely produce a geometric mess. ComplementMe addresses these challenges by learning from the natural structure of existing models without needing explicit labels.
Methodology
The authors propose a multi-stage neural architecture that leverages the natural structure of existing 3D models through weak supervision.
1. Weak Supervision via Contact Graphs
Instead of labels, the system learns from the fact that parts in a single 3D model already "belong" together. ShapeNet models are preprocessed into "contact graphs," where nodes are components and edges represent physical proximity.
2. Joint Embedding and Retrieval
Two networks are trained simultaneously:
- Embedding Network: Maps individual 3D parts (represented as point clouds) into a low-dimensional latent space.
- Retrieval Network: Takes a partial assembly and predicts a Mixture Density Network (MDN)—a probability distribution over the latent space. This allows the model to suggest multiple, diverse "modes" of parts (e.g., wheels vs. wings) rather than a single average.
3. Contrastive Loss and Placement
To train without labels, a triplet loss is used to minimize the distance between a partial assembly and its actual complement in the latent space. Once a part is retrieved, a separate Placement Network takes the geometry of both the assembly and the new part to predict the exact $(x, y, z)$ coordinates for alignment.
Results and Evaluation
The model was trained and evaluated on ShapeNet, focusing on nine categories including Airplanes, Chairs, Lamps, and Watercraft. To evaluate performance, the authors used a subset of ShapeNet with semantic labels as a "ground truth" benchmark, even though the model never saw those labels during training.
Key Findings
- Functional Compatibility: ComplementMe achieved a mean Average Precision (mAP) of 0.81 in retrieval tasks, significantly outperforming previous state-of-the-art methods (0.55).
- Geometric Accuracy: The placement network predicted positions with very low error, allowing for nearly "snap-to-fit" assembly.
- Cross-Category Creativity: The model can suggest parts across categories—such as suggesting a watercraft body to complement airplane wings—because it recognizes geometric compatibility regardless of labels.
Discussion and Limitations
ComplementMe demonstrates that geometric context from thousands of unlabeled models is sufficient to cluster interchangeable parts. However, the system currently has limitations:
- Translation Only: The placement network assumes parts are already correctly oriented and scaled.
- Physical Constraints: The model lacks an explicit understanding of physics, occasionally resulting in "floating" parts or geometric overlaps.
- Sensitivity to Scale: Very thin or small components are sometimes ignored because they contribute less to the overall point cloud density.
Future work may incorporate part deformation (scaling/stretching) and high-level attribute controls to provide designers with even more fine-grained influence over the generative process.
Summary updated