LADIS: Language Disentanglement for 3D Shape Editing

Ian Huang, Panos Achlioptas, Tianyi Zhang, Sergey Tulyakov, Minhyuk Sung, Leonidas J. Guibas

Introduction

Democratizing 3D design through natural language is a major goal in computer vision. However, current models often struggle with "entanglement." For instance, an instruction to make a chair's legs thinner might inadvertently alter the backrest or add unwanted structural elements. Previous state-of-the-art methods, such as ChangeIt3D, often became overly conservative to avoid these artifacts, resulting in edits so subtle they were barely perceptible.

LADIS produces localized, visually obvious edits compared to prior work, which often introduces entangled artifacts in irrelevant parts.

LADIS (Language Disentanglement for 3D Shape Editing) addresses this by grounding language in 3D geometry such that independent semantic concepts are represented independently in the model's latent space. This allows for localized, obvious, and semantically accurate edits without requiring explicit part-based supervision.

Methodology

The LADIS framework employs a three-pronged approach to ensure precise shape manipulation:

  1. LADIS Loss (Language Disentanglement): This regularization loss forces the embeddings of independent edit instructions to be orthogonal. By ensuring that descriptions of different parts or attributes do not overlap in the latent space, the model naturally learns to produce geometrically local edits.
  2. Multi-Expert Shape-Difference Encoder: The architecture utilizes multiple "expert" modules specializing in different geometric features (e.g., dimensions, texture, structure). A voting network dynamically selects the most relevant experts for a given text prompt.
  3. Iterative Editing Procedure:
    • Neighborhood Simplex Editing (NSE): To maintain object realism, the search space is restricted to a manifold defined by the nearest neighbors of the original shape.
  4. Output-Driven Edit Step-Size Adjustment (ODESSA): This mechanism automatically rescales optimization steps to ensure every edit results in a consistent, noticeable visual change.
The iterative editing pipeline optimizes the source shape's latent representation to maximize alignment with the edit description while maintaining realism.

Evaluation and Results

LADIS was evaluated using the ShapeTalk and ShapeGlot datasets. The model demonstrated significant improvements across several metrics:

  • Edit Locality: LADIS outperformed previous state-of-the-art methods by 20% in Part-wise Edit Precision (PEP).
  • Accuracy: It achieved a 6.6% improvement in language reference resolution.
  • Human Perception: In a study involving 500 annotations, humans rated LADIS edits as 72% more semantically accurate and 61% more localized than previous leading methods.

Qualitative results confirm that LADIS successfully produces "obvious" changes—such as clearly thinner legs—while leaving irrelevant parts of the object completely untouched.

Conclusion

By focusing on language disentanglement, LADIS provides a robust solution for localized 3D shape editing. The model moves beyond the trade-off between accuracy and visibility, enabling precise, text-driven design. Future work will look toward expanding the linguistic scope beyond English and applying these techniques to higher-fidelity 3D meshes and complex modalities.

Summary updated

Abstract

Natural language interaction is a promising direction for democratizing 3D shape design. However, existing methods for text-driven 3D shape editing face challenges in producing decoupled, local edits to 3D shapes. We address this problem by learning disentangled latent representations that ground language in 3D geometry. To this end, we propose a complementary tool set including a novel network architecture, a disentanglement loss, and a new editing procedure. Additionally, to measure edit locality, we define a new metric that we call part-wise edit precision. We show that our method outperforms existing SOTA methods by 20% in terms of edit locality, and up to 6.6% in terms of language reference resolution accuracy. Our work suggests that by solely disentangling language representations, downstream 3D shape editing can become more local to relevant parts, even if the model was never given explicit part-based supervision.
BibTeX
@inproceedings{huang2022ladis,
  author = {Ian Huang and Panos Achlioptas and Tianyi Zhang and Sergey Tulyakov and Minhyuk Sung and Leonidas J. Guibas},
  title = {{LADIS: Language Disentanglement for 3D Shape Editing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.404},
}
Figure