Introduction
Creating a 3D digital avatar that looks like a Disney princess is a popular application of AI; however, creating one that looks like a Disney princess while simultaneously following specific text instructions—such as "wearing sunglasses" or "smiling"—remains a significant technical challenge. While 2D image editing has seen massive strides, translating these multi-modal capabilities into the 3D domain often results in geometric distortions or a loss of stylistic fidelity.
Researchers from Tsinghua University have introduced ITportrait, a multi-modal framework designed for 3D portrait domain adaptation. ITportrait allows users to blend specific artistic styles from reference images with flexible, text-based editing, ensuring high-quality and 3D-consistent results.
The Challenge of Multi-modal 3D Adaptation
The primary goal of domain adaptation for 3D portraits is to take a standard 3D head model and transform it to fit a new domain, such as a sketch, a caricature, or a specific cinematic style. Existing methods typically follow one of two paths:
- Vision-based Adaptation: Uses a reference image (e.g., a painting). While accurate in style, these methods are often "one-shot" and lack flexibility for further semantic changes.
- Language-based Adaptation: Uses text prompts (e.g., "wearing sunglasses"). These are flexible but often struggle to capture the precise, nuanced textures and geometries of a specific artistic style.
When attempting to combine both modalities, existing tools often conflict. Either the text edit overwrites the artistic style, or the style is so dominant that the text edit becomes invisible. Furthermore, 3D models are prone to "geometric collapse," where the face appears correct from the front but becomes distorted when viewed from different angles.
The ITportrait Framework
ITportrait addresses these issues through a two-stage alternating training strategy that couples image and text guidance.
1. Artistic Paired Transfer (APT)
To transfer a style from a single image, the model must understand the "pose" of the subject. This is difficult for artistic drawings where traditional pose estimation fails. ITportrait uses a 2D GAN (Restyle) to create a photo-realistic proxy of the art piece. By estimating the pose from this realistic proxy, the model can train the 3D generator using a "paired" dataset, learning the style without sacrificing 3D structural integrity.
2. Image-Text Embedding (ITE)
To prevent image and text guidance from conflicting in the CLIP space, the authors introduced a threshold function ($\gamma$). This function self-adaptively controls the optimization direction:
- Style Focus: If the model has not yet captured the image style sufficiently, it prioritizes image-based optimization.
- Text Focus: Once the style is established, the focus shifts to text-driven edits.
- 3D Consistency: The model employs 3D multi-view augmentation, supervising the head from multiple angles to prevent the "flat texture" effect and ensure geometric stability.
Experimental Results
ITportrait was evaluated against state-of-the-art (SOTA) baselines, including StyleGAN-NADA, JOJOGAN, StyleCLIP, and CLIPstyler.
Key Findings:
- Qualitative Superiority: Visual results demonstrate that ITportrait successfully maintains complex styles (like "Disney") while accurately applying text-based modifications (like "blue hair"). In contrast, baseline combinations often lose the primary artistic style once a text edit is applied.
- User Preference: In a quantitative user study, ITportrait ranked highest for both Image-Style consistency (4.775/5) and Text-Style consistency (4.842/5).
- Efficiency: The framework is highly efficient, requiring only approximately 8 minutes of fine-tuning on a single RTX 3090 to adapt to a new coupled domain.
Conclusion
ITportrait represents a significant step forward in 3D avatar creation, providing a robust method for coupling visual style with linguistic flexibility. By solving the "insignificant fusion" problem through alternating optimization and 2D-to-3D pose bridging, it enables the creation of stylized 3D portraits that are both semantically flexible and geometrically sound. Future work may extend this framework to full-body avatars and more complex scene-level adaptations.
Summary updated