Complementme: weakly-supervised component suggestions for 3D modeling

Minhyuk Sung, Hao Su, Vladimir G. Kim, Siddhartha Chaudhuri, Leonidas J. Guibas

Abstract

ComplementMe is a neural framework designed to automate 3D shape assembly by suggesting both the next component to add and its precise placement. Unlike previous methods that require expensive, manually labeled datasets, ComplementMe learns from unlabeled online repositories using a weakly-supervised approach. By utilizing a probabilistic retrieval model, the system effectively handles the inherent ambiguity of design, offering multiple creative paths for 3D modeling.

Introduction

Creating 3D models from scratch is a high-barrier task requiring significant expertise. "Assembly-based modeling"—where users build shapes by kit-bashing parts from existing libraries—is a popular alternative for novices. However, as libraries grow into the thousands, finding the perfect matching part and aligning it manually becomes a "needle-in-a-haystack" problem.

Existing suggestion systems typically rely on heavy supervision, requiring datasets where every part is perfectly segmented and semantically labeled. Creating these datasets is slow and expensive. Furthermore, 3D modeling is non-functional; there isn't just one "correct" part for a partial assembly. A standard neural network trying to predict a single "average" part would likely produce a geometric mess. ComplementMe addresses these challenges by learning from the natural structure of existing models without needing explicit labels.

Methodology

The authors propose a multi-stage neural architecture that leverages the natural structure of existing 3D models through weak supervision.

The ComplementMe pipeline: a partial assembly is processed by a retrieval network to find potential components in an embedding space, which are then positioned by a placement network.

1. Weak Supervision via Contact Graphs

Instead of labels, the system learns from the fact that parts in a single 3D model already "belong" together. ShapeNet models are preprocessed into "contact graphs," where nodes are components and edges represent physical proximity.

2. Joint Embedding and Retrieval

Two networks are trained simultaneously:

  • Embedding Network: Maps individual 3D parts (represented as point clouds) into a low-dimensional latent space.
  • Retrieval Network: Takes a partial assembly and predicts a Mixture Density Network (MDN)—a probability distribution over the latent space. This allows the model to suggest multiple, diverse "modes" of parts (e.g., wheels vs. wings) rather than a single average.

3. Contrastive Loss and Placement

To train without labels, a triplet loss is used to minimize the distance between a partial assembly and its actual complement in the latent space. Once a part is retrieved, a separate Placement Network takes the geometry of both the assembly and the new part to predict the exact $(x, y, z)$ coordinates for alignment.

Results and Evaluation

The model was trained and evaluated on ShapeNet, focusing on nine categories including Airplanes, Chairs, Lamps, and Watercraft. To evaluate performance, the authors used a subset of ShapeNet with semantic labels as a "ground truth" benchmark, even though the model never saw those labels during training.

Examples of iterative assembly sequences for various categories, including chairs, tables, lamps, and watercraft.

Key Findings

  • Functional Compatibility: ComplementMe achieved a mean Average Precision (mAP) of 0.81 in retrieval tasks, significantly outperforming previous state-of-the-art methods (0.55).
  • Geometric Accuracy: The placement network predicted positions with very low error, allowing for nearly "snap-to-fit" assembly.
  • Cross-Category Creativity: The model can suggest parts across categories—such as suggesting a watercraft body to complement airplane wings—because it recognizes geometric compatibility regardless of labels.

Discussion and Limitations

ComplementMe demonstrates that geometric context from thousands of unlabeled models is sufficient to cluster interchangeable parts. However, the system currently has limitations:

  • Translation Only: The placement network assumes parts are already correctly oriented and scaled.
  • Physical Constraints: The model lacks an explicit understanding of physics, occasionally resulting in "floating" parts or geometric overlaps.
  • Sensitivity to Scale: Very thin or small components are sometimes ignored because they contribute less to the overall point cloud density.

Future work may incorporate part deformation (scaling/stretching) and high-level attribute controls to provide designers with even more fine-grained influence over the generative process.

Summary updated

Abstract

Assembly-based tools provide a powerful modeling paradigm for non-expert shape designers. However, choosing a component from a large shape repository and aligning it to a partial assembly can become a daunting task. In this paper we describe novel neural network architectures for suggesting complementary components and their placement for an incomplete 3D part assembly. Unlike most existing techniques, our networks are trained on unlabeled data obtained from public online repositories, and do not rely on consistent part segmentations or labels. Absence of labels poses a challenge in indexing the database of parts for the retrieval. We address it by jointly training embedding and retrieval networks, where the first indexes parts by mapping them to a low-dimensional feature space, and the second maps partial assemblies to appropriate complements. The combinatorial nature of part arrangements poses another challenge, since the retrieval network is not a function: several complements can be appropriate for the same input. Thus, instead of predicting a single output, we train our network to predict a probability distribution over the space of part embeddings. This allows our method to deal with ambiguities and naturally enables a UI that seamlessly integrates user preferences into the design process. We demonstrate that our method can be used to design complex shapes with minimal or no user input. To evaluate our approach we develop a novel benchmark for component suggestion systems demonstrating significant improvement over state-of-the-art techniques.
BibTeX
@article{sung2017complementme,
  author = {Minhyuk Sung and Hao Su and Vladimir G. Kim and Siddhartha Chaudhuri and Leonidas J. Guibas},
  title = {{Complementme: weakly-supervised component suggestions for 3D modeling}},
  journal = {ACM Trans. Graphics},
  year = {2017},
  doi = {10.1145/3130800.3130821},
}
Figure