ITportrait: Image-Text Coupled 3D Portrait Domain Adaptation

Xiangwen Deng, Yufeng Wang, Yuanhao Cai, Jingxiang Sun, Yebin Liu, Haoqian Wang

Introduction

Creating a 3D digital avatar that looks like a Disney princess is a popular application of AI; however, creating one that looks like a Disney princess while simultaneously following specific text instructions—such as "wearing sunglasses" or "smiling"—remains a significant technical challenge. While 2D image editing has seen massive strides, translating these multi-modal capabilities into the 3D domain often results in geometric distortions or a loss of stylistic fidelity.

Researchers from Tsinghua University have introduced ITportrait, a multi-modal framework designed for 3D portrait domain adaptation. ITportrait allows users to blend specific artistic styles from reference images with flexible, text-based editing, ensuring high-quality and 3D-consistent results.

The Challenge of Multi-modal 3D Adaptation

The primary goal of domain adaptation for 3D portraits is to take a standard 3D head model and transform it to fit a new domain, such as a sketch, a caricature, or a specific cinematic style. Existing methods typically follow one of two paths:

  1. Vision-based Adaptation: Uses a reference image (e.g., a painting). While accurate in style, these methods are often "one-shot" and lack flexibility for further semantic changes.
  2. Language-based Adaptation: Uses text prompts (e.g., "wearing sunglasses"). These are flexible but often struggle to capture the precise, nuanced textures and geometries of a specific artistic style.
Input content and style references for ITportrait, demonstrating the multi-modal nature of the task.

When attempting to combine both modalities, existing tools often conflict. Either the text edit overwrites the artistic style, or the style is so dominant that the text edit becomes invisible. Furthermore, 3D models are prone to "geometric collapse," where the face appears correct from the front but becomes distorted when viewed from different angles.

The ITportrait Framework

ITportrait addresses these issues through a two-stage alternating training strategy that couples image and text guidance.

1. Artistic Paired Transfer (APT)

To transfer a style from a single image, the model must understand the "pose" of the subject. This is difficult for artistic drawings where traditional pose estimation fails. ITportrait uses a 2D GAN (Restyle) to create a photo-realistic proxy of the art piece. By estimating the pose from this realistic proxy, the model can train the 3D generator using a "paired" dataset, learning the style without sacrificing 3D structural integrity.

2. Image-Text Embedding (ITE)

To prevent image and text guidance from conflicting in the CLIP space, the authors introduced a threshold function ($\gamma$). This function self-adaptively controls the optimization direction:

  • Style Focus: If the model has not yet captured the image style sufficiently, it prioritizes image-based optimization.
  • Text Focus: Once the style is established, the focus shifts to text-driven edits.
  • 3D Consistency: The model employs 3D multi-view augmentation, supervising the head from multiple angles to prevent the "flat texture" effect and ensure geometric stability.

Experimental Results

ITportrait was evaluated against state-of-the-art (SOTA) baselines, including StyleGAN-NADA, JOJOGAN, StyleCLIP, and CLIPstyler.

Comparison of existing image-text fusion baselines showing the difficulty of maintaining both style and text instructions simultaneously.

Key Findings:

  • Qualitative Superiority: Visual results demonstrate that ITportrait successfully maintains complex styles (like "Disney") while accurately applying text-based modifications (like "blue hair"). In contrast, baseline combinations often lose the primary artistic style once a text edit is applied.
  • User Preference: In a quantitative user study, ITportrait ranked highest for both Image-Style consistency (4.775/5) and Text-Style consistency (4.842/5).
  • Efficiency: The framework is highly efficient, requiring only approximately 8 minutes of fine-tuning on a single RTX 3090 to adapt to a new coupled domain.

Conclusion

ITportrait represents a significant step forward in 3D avatar creation, providing a robust method for coupling visual style with linguistic flexibility. By solving the "insignificant fusion" problem through alternating optimization and 2D-to-3D pose bridging, it enables the creation of stylized 3D portraits that are both semantically flexible and geometrically sound. Future work may extend this framework to full-body avatars and more complex scene-level adaptations.

Summary updated

Abstract

Domain adaptation of 3D portraits has gained more and more attention. However, the transfer mechanism of existing methods is mainly based on vision or language, which ignores the potential of vision-language combined guidance. In this paper, we propose an Image-Text multi-modal framework, namely Image and Text portrait (ITportrait), for 3D portrait domain adaptation. ITportrait relies on a two-stage alternating training strategy. In the first stage, we employ a 3D Artistic Paired Transfer (APT) method for image-guided style transfer. APT constructs paired photo-realistic portraits to obtain accurate artistic poses, which helps ITportrait to achieve high-quality 3D style transfer. In the second stage, we propose a 3D Image-Text Embedding (ITE) approach in the CLIP space. ITE uses a threshold function to self-adaptively control the optimization direction of images or texts in the CLIP space. Comprehensive experiments prove that our ITportrait achieves state-of-the-art (SOTA) results and benefits downstream tasks. All source codes and pre-trained models will be released to the public.
BibTeX
@inproceedings{deng2024itportrait,
  author = {Xiangwen Deng and Yufeng Wang and Yuanhao Cai and Jingxiang Sun and Yebin Liu and Haoqian Wang},
  title = {{ITportrait: Image-Text Coupled 3D Portrait Domain Adaptation}},
  booktitle = {ICME},
  year = {2024},
  doi = {10.1109/icme57554.2024.10687744},
}
Figure