LADIS: Language Disentanglement for 3D Shape Editing

Ian Huang, Panos Achlioptas, Tianyi Zhang, Sergey Tulyakov, Minhyuk Sung, Leonidas J. Guibas

Abstract

Editing 3D shapes using natural language is a transformative goal for democratizing 3D design, yet existing methods often struggle with "entanglement," where modifying one part of an object causes unintended changes elsewhere. This paper introduces LADIS (Language Disentanglement for 3D Shape Editing), a framework that ensures text-driven modifications remain localized to the specific components mentioned in a prompt. By regularizing the language-shape latent space and employing a neighborhood-constrained optimization strategy, LADIS achieves a 20% improvement in edit locality over previous state-of-the-art methods while producing significantly more visually distinct results.

Introduction

The ability to modify 3D assets through simple text instructions—such as "make the chair legs thinner"—is essential for making 3D modeling accessible to non-experts. However, current generative models face a dual challenge. First, they often suffer from entanglement, where a request to change a specific part (e.g., the legs) results in spurious artifacts in unrelated areas (e.g., the backrest). Second, to avoid these artifacts, many existing models become overly conservative, producing edits so subtle they are virtually imperceptible.

LADIS addresses this tension by focusing on locality: the ability to make significant, visually obvious changes to requested parts while leaving the rest of the geometry untouched.

Methodology

The LADIS framework consists of a two-stage approach: a novel training objective for representation learning and a refined iterative editing procedure.

1. Grounded Representation Learning

To achieve better control, the authors introduce the LADIS Loss (Language Disentanglement). This loss function forces the model to represent independent text instructions (e.g., "thinner legs" vs. "rungs in the back") as orthogonal vectors within a joint latent space. By ensuring that the language representations are disentangled, the model naturally learns to decouple the resulting geometric edits.

Furthermore, the architecture utilizes a Multi-Expert Encoder. Rather than relying on a single monolithic network, the system employs specialized "experts" that focus on different geometric features, such as dimensions or structural components. A voting network dynamically determines which experts to prioritize based on the specific text prompt.

2. Iterative Shape Editing

During the editing phase, LADIS employs two key mechanisms to ensure quality and visibility:

  • Neighborhood Simplex Editing (NSE): To prevent the 3D shape from degrading into unrealistic geometry during optimization, NSE constrains the edit to remain within a "neighborhood" of valid shapes derived from the training set. This ensures the output remains a plausible object.
  • ODESSA (Output-Driven Edit Step-Size Adjustment): This mechanism automatically rescales optimization steps to ensure that every latent space adjustment results in a visually perceptible change in the 3D output, effectively solving the "too subtle" problem found in previous models.

Experimental Setup and Datasets

The model was trained and evaluated using three primary datasets:

  • ShapeTalk: A large-scale dataset of 3D shape pairs and human-provided descriptions of their differences.
  • ShapeGlot: Used to evaluate the model's ability to resolve complex referential language.
  • PartNet: Provided ground-truth part segmentations to facilitate the evaluation of the new locality metrics.

Key Insights

A primary finding of this research is that regularizing language representations to be independent directly translates to geometric locality, even without explicit part-based supervision. This suggests that the structure of the language latent space is a powerful proxy for physical object decomposition. Additionally, the paper demonstrates that high locality and high visual impact are not mutually exclusive; LADIS proves that models can produce large volumetric changes without sacrificing precision.

Results

LADIS demonstrates significant improvements across both quantitative and qualitative benchmarks:

  • Edit Locality: The model outperformed the previous state-of-the-art, ChangeIt3D, by 20% in Part-wise Edit Precision (PEP), a metric designed to measure the ratio of volume change in relevant versus irrelevant parts.
  • Visual Impact: Edits produced by LADIS were volumetrically 3x larger than those of the baseline, making them significantly more visible to the user.
  • Human Evaluation: In a study involving 500 annotations, human participants judged LADIS edits to be 72% more semantically accurate and 61% more localized than the competition.
  • Language Resolution: The model also improved the accuracy of identifying target shapes from text descriptions by up to 6.6%.

Limitations and Future Work

While LADIS represents a significant step forward, it is currently limited to English-language prompts. Future research is needed to expand the framework to other spoken and sign languages. Additionally, while the method is representation-agnostic, current experiments have focused primarily on rigid categories like chairs and cars; extending these capabilities to more complex or organic shapes remains an open challenge. Finally, the effectiveness of the LADIS loss is sensitive to the quality of "negative set mining" used to identify independent instructions during training.

Summary updated

BibTeX
@inproceedings{huang2022ladis,
  author = {Ian Huang and Panos Achlioptas and Tianyi Zhang and Sergey Tulyakov and Minhyuk Sung and Leonidas J. Guibas},
  title = {{LADIS: Language Disentanglement for 3D Shape Editing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.404},
}
Figure