Complementme: weakly-supervised component suggestions for 3D modeling

Minhyuk Sung, Hao Su, Vladimir G. Kim, Siddhartha Chaudhuri, Leonidas J. Guibas

Abstract

ComplementMe is a neural framework designed to automate 3D shape assembly by suggesting both the next component to add and its precise placement. Unlike previous methods that require expensive, manually labeled datasets, ComplementMe learns from unlabeled online repositories using a weakly-supervised approach. By utilizing a probabilistic retrieval model, the system effectively handles the inherent ambiguity of design, offering multiple creative paths for 3D modeling.

Introduction

Creating 3D models from scratch is a high-barrier task requiring significant expertise. "Assembly-based modeling"—where users build shapes by kit-bashing parts from existing libraries—is a popular alternative for novices. However, as libraries grow into the thousands, finding the perfect matching part and aligning it manually becomes a "needle-in-a-haystack" problem.

Existing suggestion systems typically rely on heavy supervision, requiring datasets where every part is perfectly segmented and semantically labeled. Creating these datasets is slow and expensive. Furthermore, 3D modeling is non-functional; there isn't just one "correct" part for a partial assembly. A standard neural network trying to predict a single "average" part would likely produce a geometric mess. ComplementMe addresses these challenges by learning from the natural structure of existing models without needing explicit labels.

Methodology

The authors propose a multi-stage neural architecture that leverages the natural structure of existing 3D models through weak supervision.

The ComplementMe pipeline: a partial assembly is processed by a retrieval network to find potential components in an embedding space, which are then positioned by a placement network.

1. Weak Supervision via Contact Graphs

Instead of labels, the system learns from the fact that parts in a single 3D model already "belong" together. ShapeNet models are preprocessed into "contact graphs," where nodes are components and edges represent physical proximity.

2. Joint Embedding and Retrieval

Two networks are trained simultaneously:

  • Embedding Network: Maps individual 3D parts (represented as point clouds) into a low-dimensional latent space.
  • Retrieval Network: Takes a partial assembly and predicts a Mixture Density Network (MDN)—a probability distribution over the latent space. This allows the model to suggest multiple, diverse "modes" of parts (e.g., wheels vs. wings) rather than a single average.

3. Contrastive Loss and Placement

To train without labels, a triplet loss is used to minimize the distance between a partial assembly and its actual complement in the latent space. Once a part is retrieved, a separate Placement Network takes the geometry of both the assembly and the new part to predict the exact $(x, y, z)$ coordinates for alignment.

Results and Evaluation

The model was trained and evaluated on ShapeNet, focusing on nine categories including Airplanes, Chairs, Lamps, and Watercraft. To evaluate performance, the authors used a subset of ShapeNet with semantic labels as a "ground truth" benchmark, even though the model never saw those labels during training.

Examples of iterative assembly sequences for various categories, including chairs, tables, lamps, and watercraft.

Key Findings

  • Functional Compatibility: ComplementMe achieved a mean Average Precision (mAP) of 0.81 in retrieval tasks, significantly outperforming previous state-of-the-art methods (0.55).
  • Geometric Accuracy: The placement network predicted positions with very low error, allowing for nearly "snap-to-fit" assembly.
  • Cross-Category Creativity: The model can suggest parts across categories—such as suggesting a watercraft body to complement airplane wings—because it recognizes geometric compatibility regardless of labels.

Discussion and Limitations

ComplementMe demonstrates that geometric context from thousands of unlabeled models is sufficient to cluster interchangeable parts. However, the system currently has limitations:

  • Translation Only: The placement network assumes parts are already correctly oriented and scaled.
  • Physical Constraints: The model lacks an explicit understanding of physics, occasionally resulting in "floating" parts or geometric overlaps.
  • Sensitivity to Scale: Very thin or small components are sometimes ignored because they contribute less to the overall point cloud density.

Future work may incorporate part deformation (scaling/stretching) and high-level attribute controls to provide designers with even more fine-grained influence over the generative process.

Summary updated

BibTeX
@article{sung2017complementme,
  author = {Minhyuk Sung and Hao Su and Vladimir G. Kim and Siddhartha Chaudhuri and Leonidas J. Guibas},
  title = {{Complementme: weakly-supervised component suggestions for 3D modeling}},
  journal = {ACM Trans. Graphics},
  year = {2017},
  doi = {10.1145/3130800.3130821},
}

Loading related content...

Figure