DCT-net: domain-calibrated translation for portrait stylization

Yifang Men, Yuan Yao, Miaomiao Cui, Zhouhui Lian, Xuansong Xie

Abstract

Portrait stylization aims to transform real-world facial images into artistic domains such as 3D cartoons or anime. While recent few-shot methods leverage pre-trained generative models to mitigate data scarcity, they often suffer from significant content loss and a lack of geometric flexibility. This paper presents DCT-Net, a "calibration-first" framework designed for high-quality portrait stylization using as few as 100 style examples. By synthesizing a balanced training set through a Content Calibration Network and expanding geometric robustness, DCT-Net achieves superior identity preservation and demonstrates a unique ability to generalize from headshots to full-body stylization.

1. Introduction

The task of portrait stylization—converting a standard photograph into a stylized artistic representation—is traditionally hampered by a "data hunger" problem. High-quality results typically require thousands of paired images (original vs. cartoon), which are labor-intensive to produce. Recent advancements have attempted to solve this using few-shot fine-tuning of models like StyleGAN. However, these approaches frequently encounter two primary obstacles: content loss and structural rigidity.

Content loss occurs because fine-grained details, such as jewelry or complex backgrounds, are often lost in the model's compressed latent space. Structural rigidity refers to the limitation where models only function on perfectly aligned, front-facing headshots. DCT-Net addresses these challenges by shifting the focus from direct mapping to a domain-calibrated translation approach.

2. Methodology

DCT-Net utilizes a three-stage pipeline centered on the philosophy of "calibration first, translation later."

2.1 Content Calibration Network (CCN)

To avoid the pitfalls of sparse data, the authors employ a pre-trained source generator (trained on real faces) and a target generator adapted to the desired style. By processing identical noise vectors through both generators, the system produces perfectly aligned synthetic pairs: one real face and one stylized counterpart. This process creates a massive, balanced dataset that effectively calibrates the target style's distribution against the source domain.

2.2 Geometry Expansion Module (GEM)

To overcome the requirement for perfectly aligned faces, the Geometry Expansion Module introduces random rotations and scaling to the synthetic pairs. This teaches the model that the stylistic transformation should be independent of specific pixel coordinates, allowing the network to handle varied poses and compositions.

2.3 Texture Translation Network (TTN)

The final engine is a U-Net-based Texture Translation Network. Unlike global latent-space mapping, the U-Net focuses on local textures, which preserves "out-of-distribution" elements like glasses, hats, and intricate backgrounds. An auxiliary Facial Perception Loss is integrated via a regressor to monitor facial expressions, ensuring that while features like eyes may be exaggerated (as in anime styles), the underlying emotion and identity remain intact.

3. Experimental Results

3.1 Datasets and Evaluation

The model was trained using 10,000 images from the FFHQ dataset as source data and small exemplar sets of approximately 100 images for various target styles (3D cartoon, hand-drawn, and anime). Evaluation was conducted on 5,000 images from the CelebA dataset to test performance on diverse, real-world faces.

3.2 Quantitative and Qualitative Performance

DCT-Net demonstrated significant improvements over existing state-of-the-art methods:

  • Frechet Inception Distance (FID): DCT-Net achieved an FID of 35.92, outperforming Toonify (55.27) and CycleGAN (57.08).
  • Identity Preservation: The model led in ID preservation scores, maintaining recognizable features even under heavy stylization.
  • User Preference: In a user study, over 80% of participants preferred DCT-Net’s output, noting fewer artifacts and higher faithfulness to the original subject.

3.3 Robustness and Generalization

A key finding of the research is the model's ability to handle "hard cases," such as subjects with heavy makeup, sunglasses, or occlusions. Most notably, despite being trained primarily on headshots, the combination of the U-Net architecture and the Geometry Expansion Module allows the model to stylize full-body images in a single pass, treating the body and background as extended context for the stylized head.

4. Discussion and Limitations

DCT-Net avoids the common pitfalls of "GAN Inversion," which is often lossy, by training on synthetic pairs where the underlying code is known. However, the model is currently domain-dependent; it relies on a pre-trained source generator (FFHQ), meaning it may not translate as effectively to non-human subjects without a different base model. Additionally, while the model handles facial deformations well, extremely abstract styles (e.g., cubism) may still challenge the structural integrity of the U-Net translation. Future work may focus on calibrating clothing-specific distributions to further enhance full-body stylization.

5. Conclusion

DCT-Net introduces a robust framework for few-shot portrait stylization that prioritizes domain calibration over simple fine-tuning. By leveraging synthetic data generation and local texture translation, it provides a scalable solution that preserves identity and handles complex real-world imagery better than previous latent-space-based methods.

Summary updated

BibTeX
@article{men2022dct,
  author = {Yifang Men and Yuan Yao and Miaomiao Cui and Zhouhui Lian and Xuansong Xie},
  title = {{DCT-net: domain-calibrated translation for portrait stylization}},
  journal = {ACM Trans. Graphics},
  year = {2022},
  doi = {10.1145/3528223.3530159},
}
Figure