Material Magic Wand: Material-Aware Grouping of 3D Parts in Untextured Meshes

Umangi Jain, Vladimir Kim, Matheus Gadelha, Igor Gilitschenski, Zhiqin Chen

TL;DR

The "Material Magic Wand" is a new AI-powered tool that allows 3D artists to select all parts of a mesh that share a material with just a single click. By learning "material-aware" embeddings that combine local geometry with global context, the system can group non-identical but related parts (like all the different scales on a pinecone) far more accurately than traditional symmetry or shape-matching algorithms.

What Problem Does This Paper Address?

In professional 3D modeling, assigning materials to complex meshes is a notorious time-sink. Imagine a 3D model of a Gothic cathedral or a pinecone; these objects contain hundreds of small, repeated parts. While these parts share the same material, they are rarely identical—they vary in scale, orientation, and slight geometric deformation.

Existing tools fail here:

  1. Manual Selection: Selecting hundreds of window panes or scales one by one is tedious.
  2. Symmetry Detection: Usually looks for exact or near-isometric matches, failing when parts are structurally similar but geometrically varied.
  3. Semantic Segmentation: Groups parts by "what they are" (e.g., "leg"), but doesn't necessarily capture "what they are made of" (e.g., "chrome" vs. "plastic").

The authors bridge this gap by introducing material-aware part grouping, treating material assignment as a retrieval problem within a single 3D mesh.

Methods

The core of the system is a part encoder that maps 3D parts into a latent embedding space where parts sharing a material are clustered together.

  • Multi-Context Rendering: To understand a part, the model needs more than just its shape. It renders each part from three perspectives:
  • Isolated-part: Just the geometry of the part itself.
  • Part-with-context: The part highlighted within its immediate neighborhood.
  • Full-object: The entire mesh with the part highlighted.
  • The Architecture: These three images are fed into a vision foundation model (DINO-v3 small). The resulting features are concatenated and passed through a projection head to create a 128-dimensional embedding.
  • Supervised Contrastive Learning: The model is trained using a "Pull/Push" strategy. Embeddings of parts known to share a material are pulled together, while parts with different materials are pushed apart.
  • Interactive Inference: At runtime, a user clicks a part. The system calculates the $L_1$ distance between that part’s embedding and all others in the mesh. A "tolerance" parameter (similar to Photoshop’s Magic Wand) allows the user to expand or contract the selection based on similarity.

Datasets

Training a model like this requires massive amounts of labeled 3D data. The authors curated a subset of Objaverse, resulting in approximately 1.9 million parts across 22,000 meshes with reliable material annotations.

Because raw 3D data is often "noisy" (e.g., a designer might accidentally give two identical fence slats different material IDs), the authors also introduced a new curated benchmark. This benchmark consists of 100 complex shapes with 241 manually refined part-level queries, providing a "gold standard" for evaluating material-aware grouping.

Key Insights and Ideas

  • Context is King: One of the paper's biggest findings is that geometry alone isn't enough to predict material. A small cylinder might be a metal bolt on a machine or a wooden peg on a chair. By including "part-with-context" and "full-object" views, the model learns to use the part's environment to disambiguate its material.
  • Deduplication for Speed: Real-world meshes often have thousands of parts. The authors use a histogram-matching algorithm to identify perfectly identical parts first. This "deduplication" step speeds up inference by up to 520x without sacrificing much accuracy.
  • Embeddings over Labels: Instead of training a classifier to say "this is wood," the model learns a relative similarity space. This allows it to work on materials it has never seen before, as long as they follow the geometric and contextual patterns it learned during training.

Key Results

The Material Magic Wand significantly outperforms existing baselines across all metrics:

  • Accuracy: It achieved an AUC PR of 89.74%, compared to 81.14% for the strongest baseline (DINO-v3) and a meager 26.85% for traditional geometric histogram matching.
  • Grouping Quality: In terms of F1 score (a balance of precision and recall), the method scored 75.94%, a +16.6% improvement over the best existing vision-based models.
  • Qualitative Success: The model successfully groups complex, non-identical structures like the varied leaves of a tree, the different wheels of a locomotive, and the intricate jewels on a crown—tasks where previous methods typically failed.

Limitations and Future Work

Despite its strengths, the "Wand" isn't perfect:

  • Extreme Variation: If parts share a material but look completely different (e.g., a knotted cable vs. a straight cable), the model may miss the connection.
  • Geometric Mimicry: If two different materials are used on identical geometries (like black and white pieces in a chess set), the model will incorrectly group them because it cannot "see" color or texture on these untextured meshes.
  • Artistic Intent: Sometimes material assignment is a stylistic choice by the artist that doesn't follow geometric logic. The authors suggest that future work could involve "Multiple Clicks" to help the model learn specific user intents on the fly.

Summary updated

Abstract

We introduce the problem of material-aware part grouping in untextured meshes. Many real-world shapes, such as scales of pinecones or windows of buildings, contain repeated structures that share the same material but exhibit geometric variations. When assigning materials to such meshes, these repeated parts often require piece-by-piece manual identification and selection, which is tedious and time-consuming. To address this, we propose Material Magic Wand, a tool that allows artists to select part groups based on their estimated material properties -- when one part is selected, our algorithm automatically retrieves all other parts likely to share the same material. The key component of our approach is a part encoder that generates a material-aware embedding for each 3D part, accounting for both local geometry and global context. We train our model with a supervised contrastive loss that brings embeddings of material-consistent parts closer while separating those of different materials; therefore, part grouping can be achieved by retrieving embeddings that are close to the embedding of the selected part. To benchmark this task, we introduce a curated dataset of 100 shapes with 241 part-level queries. We verify the effectiveness of our method through extensive experiments and demonstrate its practical value in an interactive material assignment application.
BibTeX
@inproceedings{jain2026material,
  author = {Umangi Jain and Vladimir Kim and Matheus Gadelha and Igor Gilitschenski and Zhiqin Chen},
  title = {{Material Magic Wand: Material-Aware Grouping of 3D Parts in Untextured Meshes}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=Imj0axc0Bf},
}
Figure