Abstract
Plasticine3D is a novel framework designed for text-guided non-rigid editing of 3D objects. Unlike existing methods that struggle with significant structural deformations or identity loss, Plasticine3D enables complex changes—such as altering a character's pose—while meticulously preserving the original object's identity. The framework introduces a three-phase Score Projection Sampling (SPS) strategy, Multi-View-Embedding (MVE) optimization, and Embedding-Fusion (EF) to overcome common pitfalls like detail loss and geometric instability (the "Janus problem"). Experimental results demonstrate that Plasticine3D outperforms state-of-the-art baselines in both qualitative fidelity and quantitative CLIP similarity metrics.
Introduction
The field of 3D object editing via text prompts has advanced rapidly, leveraging Score Distillation Sampling (SDS) and 2D diffusion priors. However, a significant gap remains between "rigid" editing (texture changes) and "non-rigid" editing (structural or postural changes). Non-rigid editing faces two primary challenges: maintaining identity during large deformations and ensuring geometric consistency across views—avoiding the "Janus problem" where models exhibit multiple faces or inconsistent geometry.
Existing tools often produce low-quality results where fine-grained details are washed out or the object's original character is lost. Plasticine3D addresses these issues by decoupling geometry and texture editing and introducing a robust view-dependent guidance system that allows for significant structural shifts without compromising the source model's integrity.
Methodology
Plasticine3D utilizes a two-stage pipeline consisting of geometry editing followed by texture refinement. Three core technical innovations drive the framework:
- Multi-View-Embedding (MVE) Optimization: To capture the complexity of 3D objects, the system optimizes a set of trainable embeddings for different viewpoints. Linear interpolation between these embeddings provides view-dependent guidance, ensuring the model recognizes the object's features from all angles.
- Embedding-Fusion (EF): This allows for fine-grained control by blending optimized "original" embeddings with "target" text embeddings. A fusion rate ($r$) enables users to balance identity preservation with the desired structural change.
- Score Projection Sampling (SPS): Replacing standard SDS, SPS operates in three phases: Target Enhancement (prioritizing structural change), Fused Editing (aligning with the blended goal), and Detail Enhancement (reinforcing intricate details post-deformation).
Additionally, the framework incorporates multi-view normal-depth guidance using models like RichDreamer to act as a geometric regulator, preventing physical implausibility and ensuring the edited mesh remains consistent from all perspectives.
Results and Discussion
Evaluated using the Objaverse dataset, Plasticine3D was compared against state-of-the-art baselines including Vox-E and DreamBooth3D. Quantitatively, it achieved superior performance with a CLIP similarity score ($CLIP_{sim}$) of 0.295 and a directional similarity ($CLIP_{dir}$) of 0.0456, indicating that the edits were both high-quality and highly faithful to the text prompts.
Qualitatively, visual results show the model successfully executing complex non-rigid tasks, such as transforming a sitting cat into a standing one or a static Ironman into a running pose, while preserving specific armor patterns, colors, and the overall "feel" of the original assets. A key insight from this work is that a single CLIP embedding is insufficient for 3D complexity; view-dependent embeddings are crucial for identity preservation. Furthermore, the use of gradient projection in SPS allows for feature enhancement without introducing noise or muddying the semantic target.
Conclusion
Plasticine3D represents a significant step forward in non-rigid 3D editing. By addressing the limitations of single-view embeddings and standard SDS, it provides a robust solution for high-fidelity structural manipulation. While the framework currently faces hurdles regarding computational hardware requirements and slight scale shifts during geometry editing, it sets a new standard for identity-preserving 3D deformation. Future work will focus on optimizing the UNet fine-tuning step and exploring tuning-free editing methods using control points or skeletons for even more precise structural manipulation.
Summary updated