ITportrait: Image-Text Coupled 3D Portrait Domain Adaptation

Xiangwen Deng, Yufeng Wang, Yuanhao Cai, Jingxiang Sun, Yebin Liu, Haoqian Wang

ITportrait: Mastering the Art of 3D Portrait Adaptation with Image and Text

Creating a 3D digital avatar that looks like a Disney princess is cool; creating one that looks like a Disney princess wearing sunglasses and smiling is much harder. While AI has made massive strides in 2D image editing, translating those capabilities into the 3D world—specifically for portraits—remains a significant challenge. A new paper from researchers at Tsinghua University introduces ITportrait, a framework that finally allows us to blend specific artistic styles with flexible text-based editing in 3D.

TL;DR

ITportrait is a multi-modal framework for 3D portrait domain adaptation that simultaneously follows an image-based style and text-based instructions. By using a two-stage alternating training strategy and a clever "threshold function" in CLIP space, it achieves high-quality, 3D-consistent results that outperform existing vision-only or language-only methods.

What Problem Does This Paper Address?

The goal is "domain adaptation" for 3D portraits—taking a standard 3D head model and making it fit a new domain (like a sketch, a caricature, or a specific movie style).

Current methods usually pick one lane:

  1. Vision-based: You provide a reference image (e.g., a painting of the Joker), and the model copies that style. However, these are often "one-shot" and offer little flexibility for further changes.
  2. Language-based: You type "wearing sunglasses," and the model edits the 3D head. These are flexible but often struggle to capture the precise, nuanced textures of a specific artistic style.

When you try to do both at once with existing tools, they tend to fight each other. Either the text edit overwrites the artistic style, or the style is so strong the text edit becomes invisible. Furthermore, 3D models are prone to "geometric collapse"—where the face looks fine from the front but turns into a distorted mess when viewed from the side.

Methods

ITportrait solves these issues using a two-stage alternating training strategy:

1. Artistic Paired Transfer (APT)

To transfer a style from a single image, the model needs to know the "pose" of the person in that image. This is notoriously difficult for artistic drawings. The authors propose a clever workaround: they use a 2D GAN (Restyle) to create a photo-realistic version of the art piece. Since it's easier to estimate the pose of a real human face, they use the photo-realistic proxy to get the pose, then use that to train the 3D generator. This creates a "paired" dataset that helps the model learn the style without losing the 3D structure.

2. Image-Text Embedding (ITE)

This is where the "coupling" happens. The model operates in CLIP space (a shared space for images and text). To prevent the image and text guidance from conflicting, the authors introduced a threshold function ($\gamma$).

  • If the model hasn't captured the image style well enough yet, it focuses on the image.
  • Once the style is established, it shifts focus to the text-driven edits.
  • They also use 3D multi-view augmentation, supervising the model from multiple angles to ensure the head remains a solid, consistent 3D object rather than a "flat" texture.

Datasets

  • FFHQ: The model uses the Flickr-Faces-HQ dataset for its pre-trained 3D backbone (EG3D).
  • Artistic References: The authors tested against various domains including Disney, Pixar, sketches, and specific characters like the Joker.
  • Evaluation: They compared their results against SOTA baselines like StyleGAN-NADA, JOJOGAN, StyleCLIP, and CLIPstyler.

Key Insights and Ideas

  • The "Insignificant Fusion" Problem: The authors found that optimizing for image and text simultaneously often leads to mediocre results for both. Their solution—alternating between the two based on a progress threshold—is a simple but highly effective "aha moment."
  • 2D Proxies for 3D Poses: Realizing that 3D pose estimation is the bottleneck for 1-shot style transfer and solving it via a 2D photo-realistic "bridge" is a practical engineering win.
  • Multi-view is Mandatory: Text-driven 3D editing often ignores the "sides" of the head. By forcing the model to look at the head from different angles during training, they prevent the "nose collapse" common in other 3D GAN editing methods.

Key Results

  • Qualitative Superiority: Visual results show that ITportrait successfully keeps the "Disney" look while adding "blue hair" or "sunglasses," whereas baseline combinations (like JOJOGAN + StyleCLIP) often lose the Disney style entirely once the edit is applied.
  • User Study: In a quantitative user study, ITportrait was ranked highest for Image-Style (4.775/5) and Text-Style (4.842/5) consistency compared to other fusion methods.
  • Efficiency: The model is remarkably fast, requiring only about 8 minutes of fine-tuning on a single RTX 3090 to adapt to a new coupled domain.

Limitations and Future Work

While the paper demonstrates impressive results, a few areas remain for exploration:

  • Extreme Poses: The pose estimation relies on 2D-to-3D inversion, which can still struggle with very extreme artistic exaggerations.
  • CLIP Bias: Like all CLIP-based methods, the model is subject to the inherent biases and limitations of the CLIP encoders.
  • Future Directions: The authors suggest this framework could be extended to full-body avatars or more complex scene-level domain adaptations in the future.

Summary updated

Abstract

Domain adaptation of 3D portraits has gained more and more attention. However, the transfer mechanism of existing methods is mainly based on vision or language, which ignores the potential of vision-language combined guidance. In this paper, we propose an Image-Text multi-modal framework, namely Image and Text portrait (ITportrait), for 3D portrait domain adaptation. ITportrait relies on a two-stage alternating training strategy. In the first stage, we employ a 3D Artistic Paired Transfer (APT) method for image-guided style transfer. APT constructs paired photo-realistic portraits to obtain accurate artistic poses, which helps ITportrait to achieve high-quality 3D style transfer. In the second stage, we propose a 3D Image-Text Embedding (ITE) approach in the CLIP space. ITE uses a threshold function to self-adaptively control the optimization direction of images or texts in the CLIP space. Comprehensive experiments prove that our ITportrait achieves state-of-the-art (SOTA) results and benefits downstream tasks. All source codes and pre-trained models will be released to the public.
BibTeX
@inproceedings{deng2024itportrait,
  author = {Xiangwen Deng and Yufeng Wang and Yuanhao Cai and Jingxiang Sun and Yebin Liu and Haoqian Wang},
  title = {{ITportrait: Image-Text Coupled 3D Portrait Domain Adaptation}},
  booktitle = {ICME},
  year = {2024},
  doi = {10.1109/icme57554.2024.10687744},
}

Loading related content...

Figure