ShapeTalk: A Language Dataset and Framework for 3D Shape Edits and Deformations

Panos Achlioptas, Ian Huang, Minhyuk Sung, Sergey Tulyakov, Leonidas J. GuibasCVPR 2023 · OpenReview

Abstract

Editing 3D geometry is a challenging task requiring specialized skills. In this work, we aim to facilitate the task of editing the geometry of 3D models through the use of natural language. For example, we may want to modify a 3D chair model to “make its legs thinner” or to “open a hole in its back”. To tackle this problem in a manner that promotes open-ended language use and enables fine-grained shape edits, we introduce the most extensive existing corpus of natural language utterances describing shape differences: ShapeTalk. ShapeTalk contains over half a million discriminative utterances produced by contrasting the shapes of common 3D objects for a variety of object classes and degrees of similarity. We also introduce a generic framework, ChangeIt3D, which builds on ShapeTalk and can use an arbitrary 3D generative model of shapes to produce edits that align the output better with the edit or deformation description. Finally, we introduce metrics for the quantitative evaluation of language-assisted shape editing methods that reflect key desiderata within this editing setup. We note that ShapeTalk allows methods to be trained with explicit 3D-to-language data, bypassing the necessity of “lifting” 2D to 3D using methods like neural rendering, as required by extant 2D image-language foundation models. Our code and data are publicly available at https://changeit3d.github.io/.

Introduction

While text-to-image generation has seen explosive growth with models like DALL-E and Midjourney, editing 3D geometry via natural language remains a significant hurdle. 3D content creation typically requires specialized software and technical expertise. The primary challenges include the scarcity of fine-grained 3D-language data, the difficulty of reference resolution (identifying which part of a mesh to modify), and the lack of standardized evaluation metrics for geometric realism and textual faithfulness.

The ShapeTalk Dataset

To address data scarcity, the authors introduce ShapeTalk, the largest dataset of its kind. It comprises 536,596 utterances describing 36,391 shapes across 30 object classes. ShapeTalk was collected using a discriminative methodology: human annotators were shown a "target" shape and a "distractor" shape and asked to describe their differences. This approach forces the language to be highly specific (e.g., "the backrest is more curved" rather than just "it is a chair"), providing a high density of "hard" contexts where very similar objects are compared. This dataset is roughly seven times larger than previous benchmarks like ShapeGlot.

The ChangeIt3D Framework

The authors propose ChangeIt3D, a modular framework that enables users to modify 3D models using simple text prompts without requiring explicit geometric part labels. The framework operates in two stages:

  1. Representation & Listening: A 3D generative model (such as a Point-Cloud Autoencoder) learns a latent space of shapes. Simultaneously, a Neural Listener is trained to predict which shape in a pair better matches a given text description.
  2. The Shape Editor: This module learns to navigate the latent space. Given a starting shape and an edit prompt, it predicts an "update vector" that shifts the shape's position in the latent space toward a version that satisfies the prompt.
The ChangeIt3D framework architecture, illustrating the generative, discriminative, and editing pipelines.

Key innovations in ChangeIt3D include the decoupling of update direction and magnitude for improved stability, and a "self-contrast" training objective where the model is rewarded if the edited shape is more compatible with the text than the original.

Results and Evaluation

ChangeIt3D demonstrates a remarkable ability for implicit part learning. The framework learns to identify and edit specific parts (like "armrests" or "blades") purely from language, without ever being shown manual 3D segmentation masks.

Quantitatively, the Neural Listener achieved 71.3% accuracy in target identification. Using a new metric called Linguistic Association Boost (LAB), the authors show that ChangeIt3D effectively moves shapes in the latent space to match user intent while maintaining the overall identity of the original model. Qualitative results confirm that the model can perform precise, localized edits across a wide variety of object categories.

Qualitative results showing 3D shape edits across various categories, demonstrating fine-grained control over attributes like thickness, height, and part presence.

Conclusion and Future Work

ShapeTalk and ChangeIt3D provide a robust foundation for language-driven 3D editing. By training on descriptions of differences between objects, the model gains a nuanced understanding of geometry. While challenges remain—such as the inherent ambiguity of natural language and occasional "bleeding" of edits into adjacent areas—this work significantly lowers the barrier to 3D content creation. Future research will explore better disentanglement of shape attributes to ensure even cleaner, more localized edits.

Summary updated

Abstract

Editing 3D geometry is a challenging task requiring specialized skills. In this work, we aim to facilitate the task of editing the geometry of 3D models through the use of natural language. For example, we may want to modify a 3D chair model to “make its legs thinner” or to “open a hole in its back”. To tackle this problem in a manner that promotes open-ended language use and enables fine-grained shape edits, we introduce the most extensive existing corpus of natural language utterances describing shape differences: ShapeTalk. ShapeTalk contains over half a million discriminative utterances produced by contrasting the shapes of common 3D objects for a variety of object classes and degrees of similarity. We also introduce a generic framework, ChangeIt3D, which builds on ShapeTalk and can use an arbitrary 3D generative model of shapes to produce edits that align the output better with the edit or deformation description. Finally, we introduce metrics for the quantitative evaluation of language-assisted shape editing methods that reflect key desiderata within this editing setup. We note that ShapeTalk allows methods to be trained with explicit 3D-to-language data, bypassing the necessity of “lifting” 2D to 3D using methods like neural rendering, as required by extant 2D image-language foundation models. Our code and data are publicly available at https://changeit3d.github.io/.
BibTeX
@inproceedings{achlioptas2023shapetalk,
  author = {Panos Achlioptas and Ian Huang and Minhyuk Sung and Sergey Tulyakov and Leonidas J. Guibas},
  title = {{ShapeTalk: A Language Dataset and Framework for 3D Shape Edits and Deformations}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=zEOgUxmraH},
}
Figure