Plasticine3D: Non-rigid 3D editting with text guidance

Yige Chen, Ang Chen, Siyuan Chen, Ran YiarXiv 2023 · DOI · arXiv

Abstract

Plasticine3D is a novel framework designed for text-guided non-rigid editing of 3D objects. Unlike existing methods that struggle with significant structural deformations or identity loss, Plasticine3D enables complex changes—such as altering a character's pose—while meticulously preserving the original object's identity. The framework introduces a three-phase Score Projection Sampling (SPS) strategy, Multi-View-Embedding (MVE) optimization, and Embedding-Fusion (EF) to overcome common pitfalls like detail loss and geometric instability (the "Janus problem"). Experimental results demonstrate that Plasticine3D outperforms state-of-the-art baselines in both qualitative fidelity and quantitative CLIP similarity metrics.

Introduction

The field of 3D object editing via text prompts has advanced rapidly, leveraging Score Distillation Sampling (SDS) and 2D diffusion priors. However, a significant gap remains between "rigid" editing (texture changes) and "non-rigid" editing (structural or postural changes). Non-rigid editing faces two primary challenges: maintaining identity during large deformations and ensuring geometric consistency across views—avoiding the "Janus problem" where models exhibit multiple faces or inconsistent geometry.

Qualitative results of Plasticine3D showing non-rigid edits like a running ironman and a lying dog.

Existing tools often produce low-quality results where fine-grained details are washed out or the object's original character is lost. Plasticine3D addresses these issues by decoupling geometry and texture editing and introducing a robust view-dependent guidance system that allows for significant structural shifts without compromising the source model's integrity.

Methodology

Plasticine3D utilizes a two-stage pipeline consisting of geometry editing followed by texture refinement. Three core technical innovations drive the framework:

  1. Multi-View-Embedding (MVE) Optimization: To capture the complexity of 3D objects, the system optimizes a set of trainable embeddings for different viewpoints. Linear interpolation between these embeddings provides view-dependent guidance, ensuring the model recognizes the object's features from all angles.
  2. Embedding-Fusion (EF): This allows for fine-grained control by blending optimized "original" embeddings with "target" text embeddings. A fusion rate ($r$) enables users to balance identity preservation with the desired structural change.
  3. Score Projection Sampling (SPS): Replacing standard SDS, SPS operates in three phases: Target Enhancement (prioritizing structural change), Fused Editing (aligning with the blended goal), and Detail Enhancement (reinforcing intricate details post-deformation).
The Plasticine3D pipeline, illustrating Multi-view Embedding Optimization, Embedding Fusion, and Score Projection Sampling.

Additionally, the framework incorporates multi-view normal-depth guidance using models like RichDreamer to act as a geometric regulator, preventing physical implausibility and ensuring the edited mesh remains consistent from all perspectives.

Results and Discussion

Evaluated using the Objaverse dataset, Plasticine3D was compared against state-of-the-art baselines including Vox-E and DreamBooth3D. Quantitatively, it achieved superior performance with a CLIP similarity score ($CLIP_{sim}$) of 0.295 and a directional similarity ($CLIP_{dir}$) of 0.0456, indicating that the edits were both high-quality and highly faithful to the text prompts.

Qualitatively, visual results show the model successfully executing complex non-rigid tasks, such as transforming a sitting cat into a standing one or a static Ironman into a running pose, while preserving specific armor patterns, colors, and the overall "feel" of the original assets. A key insight from this work is that a single CLIP embedding is insufficient for 3D complexity; view-dependent embeddings are crucial for identity preservation. Furthermore, the use of gradient projection in SPS allows for feature enhancement without introducing noise or muddying the semantic target.

Conclusion

Plasticine3D represents a significant step forward in non-rigid 3D editing. By addressing the limitations of single-view embeddings and standard SDS, it provides a robust solution for high-fidelity structural manipulation. While the framework currently faces hurdles regarding computational hardware requirements and slight scale shifts during geometry editing, it sets a new standard for identity-preserving 3D deformation. Future work will focus on optimizing the UNet fine-tuning step and exploring tuning-free editing methods using control points or skeletons for even more precise structural manipulation.

Summary updated

Abstract

With the help of Score Distillation Sampling (SDS) and the rapid development of neural 3D representations, some methods have been proposed to perform 3D editing such as adding additional geometries, or overwriting textures. However, generalized 3D non-rigid editing task, which requires changing both the structure (posture or composition) and appearance (texture) of the original object, remains to be challenging in 3D editing field. In this paper, we propose Plasticine3D, a novel text-guided fine-grained controlled 3D editing pipeline that can perform 3D non-rigid editing with large structure deformations. Our work divides the editing process into a geometry editing stage and a texture editing stage to achieve separate control of structure and appearance. In order to maintain the details of the original object from different viewpoints, we propose a Multi-View-Embedding (MVE) Optimization strategy to ensure that the guidance model learns the features of the original object from various viewpoints. For the purpose of fine-grained control, we propose Embedding-Fusion (EF) to blend the original characteristics with the editing objectives in the embedding space, and control the extent of editing by adjusting the fusion rate. Furthermore, in order to address the issue of gradual loss of details during the generation process under high editing intensity, as well as the problem of insignificant editing effects in some scenarios, we propose Score Projection Sampling (SPS) as a replacement of score distillation sampling, which introduces additional optimization phases for editing target enhancement and original detail maintenance, leading to better editing quality. Extensive experiments demonstrate the effectiveness of our method on 3D non-rigid editing tasks
BibTeX
@misc{chen2023plasticine,
  author = {Yige Chen and Ang Chen and Siyuan Chen and Ran Yi},
  title = {{Plasticine3D: Non-rigid 3D editting with text guidance}},
  year = {2023},
  eprint = {2312.10111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.10111},
}
Figure