ShapeWalk: Compositional Shape Editing Through Language-Guided Chains

Habib Slim, Mohamed Elhoseiny

Abstract

Editing 3D shapes through natural language instructions is a challenging task that requires the comprehension of both language semantics and fine-grained geometric details. To bridge this gap, we introduce ShapeWalk, a carefully designed synthetic dataset designed to advance the field of language-guided shape editing. The dataset consists of 158K unique shapes connected through 26K edit chains, with an average length of 14 chained shapes. Each consecutive pair of shapes is associated with precise language instructions describing the applied edits. We synthesize edit chains by reconstructing and interpolating shapes sampled from a realistic CAD-designed 3D dataset in the parameter space of the GeoCode shape program. We leverage rule-based methods and language models to generate accurate and realistic natural language prompts corresponding to each edit. To illustrate the practicality of our contribution, we train neural editor modules in the latent space of shape autoencoders, and demonstrate the ability of our dataset to enable a variety of language-guided shape edits. Finally, we introduce multi-step editing metrics to benchmark the capacity of our models to perform recursive shape edits. We hope that our work will enable further study of compositional language-guided shape editing, and finds application in 3D CAD design and interactive modeling.

Introduction

Editing 3D models is traditionally a labor-intensive task requiring specialized CAD skills. While using natural language to "talk" to a 3D editor is an intuitive goal, it faces a significant data bottleneck. Existing datasets often provide pairs of shapes where a single text prompt corresponds to multiple simultaneous geometric changes. This "entanglement" makes it difficult for neural networks to learn the precise mapping between specific words and specific geometric parameters. Furthermore, most datasets focus on single-step edits, failing to address the recursive, multi-step nature of real-world design.

To address these challenges, we introduce ShapeWalk, a framework and dataset that decomposes complex 3D modifications into sequences of single-factor changes, enabling models to learn compositional, multi-step shape editing.

Overview of ShapeWalk edit chains, showing sequential parameter modifications and corresponding natural language instructions.

The ShapeWalk Framework

The core of our approach is the generation of "edit chains." Each link in a chain represents a change to a single geometric attribute, providing a clear signal for learning.

Dataset Generation Pipeline

The generation process involves three main steps:

  1. Parameterization via GeoCode: We utilize GeoCode, a procedural shape program that represents 3D objects through interpretable parameters such as seat_height, leg_thickness, or backrest_slope.
  2. Chain Generation: We take realistic CAD models from the 3DCoMPaT++ dataset and reconstruct them into the GeoCode parameter space. By interpolating parameters one by one between two distinct shapes, we create a sequence of intermediate shapes that form an "edit walk."
  3. Language Grounding: For every parameter change, we generate a rule-based description. To ensure linguistic variety and naturalness, we use a T5 transformer model to paraphrase these into diverse instructions.
The ShapeWalk generation pipeline: CAD shapes are mapped into a shape program space (GeoCode), where chains are formed through parameter interpolation.

Neural Shape Editing

We train a neural editor module—a latent mapper—that operates in the latent space of shape autoencoders (such as PC-AE or 3D2VS). The editor takes a shape's latent representation and a text embedding (from BERT) to predict an "edit vector." A key architectural insight was decoupling direction and magnitude: predicting the direction of the geometric change separately from its intensity significantly improved performance over predicting a single edit vector.

Results and Insights

Our experiments demonstrate that training on sequences allows models to learn that complex edits are compositions of simpler ones, enabling effective recursive editing.

  • Quantitative Performance: Our best model (a decoupled MLP) achieved a Chamfer Distance (CD) of $2.670 \times 10^{-4}$ on the chained editing task, outperforming more complex "Late Fusion" architectures.
  • Edit Recognition: A classifier trained to identify applied instructions achieved 100% accuracy on high-level changes like seat height. However, performance dropped to 58.7% for subtle changes like leg roundness, highlighting a bottleneck in fine-grained 3D perception.
  • The Autoencoder Bottleneck: We found that the editor's performance is often capped by the underlying shape representation. If the latent space cannot capture fine details (e.g., the number of rails in a chair's backrest), the model cannot successfully execute language-guided edits for those features.

Conclusion

ShapeWalk provides a massive, high-quality resource for training models in compositional 3D shape editing. By focusing on single-factor edit chains, we move closer to intuitive, language-guided CAD design. Future work will explore more robust shape representations, such as meshes or NeRFs, to overcome current limitations in capturing fine-grained geometric details.

Summary updated

Abstract

Editing 3D shapes through natural language instructions is a challenging task that requires the comprehension of both language semantics and fine-grained geometric details. To bridge this gap, we introduce ShapeWalk, a carefully designed synthetic dataset designed to advance the field of language-guided shape editing. The dataset consists of 158K unique shapes connected through 26K edit chains, with an average length of 14 chained shapes. Each consecutive pair of shapes is associated with precise language instructions describing the applied edits. We synthesize edit chains by reconstructing and interpolating shapes sampled from a realistic CAD-designed 3D dataset in the parameter space of the GeoCode shape program. We leverage rule-based methods and language models to generate accurate and realistic natural language prompts corresponding to each edit. To illustrate the practicality of our contribution, we train neural editor modules in the latent space of shape autoencoders, and demonstrate the ability of our dataset to enable a variety of language-guided shape edits. Finally, we introduce multi-step editing metrics to benchmark the capacity of our models to perform recursive shape edits. We hope that our work will enable further study of compositionallanguage-guided shape editing, and finds application in 3D CAD design and interactive modeling.
BibTeX
@inproceedings{slim2024shapewalk,
  author = {Habib Slim and Mohamed Elhoseiny},
  title = {{ShapeWalk: Compositional Shape Editing Through Language-Guided Chains}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=6zmie8VtiD},
}
Figure