3D-CariGAN: An End-to-End Solution to 3D Caricature Generation from Face Photos

Zipeng Ye, Ran Yi, Minjing Yu, Juyong Zhang, Yu-Kun Lai, Yong-Jin Liu

TL;DR

This paper introduces 3D-CariGAN, an end-to-end deep learning system that generates high-quality 3D caricatures directly from normal 2D face photos. It tackles the challenging cross-domain problem by introducing a novel 3D caricature PCA model and specialized loss functions that preserve identity while exaggerating features.

What Problem Does This Paper Address?

Creating 3D caricatures is an artistic process that is time-consuming and requires specialized 3D modeling tools. Existing methods for generating 3D caricatures often require input that is difficult for non-professionals to provide, such as caricature sketches or 2D caricatures. The core challenge is to bridge the significant gap between standard 2D face photos and the exaggerated, stylized geometry of 3D caricatures, while also maintaining the subject's identity. This paper aims to automate this process, making 3D caricature generation accessible from simple face photos.

Methods

The authors propose a three-step pipeline:

  1. 3D Caricature Mesh Generation (3D-CariGAN): This is the core of the system. They employ a Generative Adversarial Network (GAN) architecture. Crucially, instead of directly predicting mesh vertices (which can lead to noisy results), they train the GAN to predict parameters for a novel 3D Caricature PCA (3DCariPCA) model. This PCA model is built from a large dataset of 3D caricatures and provides a more constrained and efficient representation.
  2. Normal 3D Head Reconstruction: To establish correspondence and transfer texture, they first reconstruct a normal 3D head mesh from the input 2D photo. This step uses existing methods for 3D face reconstruction.
  3. Texture Mapping: The texture from the input photo is mapped onto the generated 3D caricature mesh. This is achieved by establishing correspondences between the normal 3D head and the 3D caricature mesh, leveraging the consistent connectivity and landmark positions.

Novel contributions include:

  • 3D Caricature PCA Model: A PCA model specifically trained on 3D caricature shapes, enabling better representation and interpolation within the caricature domain.
  • Character Loss (Lcha): A loss function designed to preserve the identity of the person between the input photo and the generated caricature, based on the cosine similarity of feature vectors derived from psychological studies.
  • Caricature Loss (Lcari): A loss function that encourages the desired exaggeration and stylistic features of a caricature, also informed by psychological studies on caricature perception.

The system also offers simple user controls to adjust the exaggeration level interactively.

Datasets

  • CelebAMask-HQ: Used for training the normal 3D head reconstruction component, containing 30,000 portrait photos.
  • 3DCari Dataset: A newly created dataset of 5,343 3D caricature meshes with consistent connectivity. This dataset was built by collecting 2D caricature images and then using an optimization-based method to reconstruct their 3D counterparts.

The training datasets are unpaired, meaning a direct 2D photo to 3D caricature correspondence is not available.

Key Insights and Ideas

  • PCA for Caricatures is Crucial: Directly predicting 3D mesh vertices is problematic due to the high dimensionality and lack of strong constraints. A PCA model specifically for caricature shapes provides a much more stable and controllable representation.
  • Bridging the Domain Gap with Intermediate Representation: Reconstructing a normal 3D head serves as a vital bridge, allowing for texture transfer and establishing correspondences between the input photo and the target 3D caricature.
  • Psychology-Informed Loss Functions: Leveraging insights from psychological studies on caricature perception (identity preservation vs. exaggeration) leads to more effective and targeted loss functions (Lcha and Lcari), which are key to achieving high-quality results.
  • End-to-End Efficiency: The proposed GAN-based approach, combined with the PCA representation, allows for efficient, end-to-end generation, significantly outperforming multi-step baseline methods.

Key Results

  • Qualitative: The system generates high-quality 3D caricatures that are visually appealing and maintain the identity of the input subject. User studies indicate the method produces caricatures with good structure, similarity, and distinctiveness.
  • Quantitative:
    • Speed: The proposed 3D-CariGAN is significantly faster than the baseline method. For example, landmark detection and warping in the baseline take 7.431s, while the entire 3D-CariGAN process takes 1.109s (on GPU).
  • User Study: In a two-level user study, 3D-CariGAN achieved the best scores for "Reasonable Structure," "Similarity," and "Distinctiveness," while the baseline performed best on "Weirdness."

Limitations and Future Work

  • Texture Generation: The current method primarily uses the texture from the input photo. Generating complete, artistically stylized textures for 3D caricatures (especially for occluded or unseen parts) remains a challenge. The authors suggest that a dataset of 3D caricatures with artist-created textures would be beneficial for future work.
  • Texture Representation: Textures are currently defined as colors at mesh vertices, which can be low-resolution. Exploring more sophisticated texture mapping techniques is suggested.
  • Identity vs. Weirdness Balance: While the method achieves a good balance, the baseline method (which uses 2D caricature information) was rated as more "weird." Future work could explore ways to further enhance the "weirdness" while maintaining identity and structure.

Summary updated

Abstract

Caricature is a type of artistic style of human faces that attracts considerable attention in the entertainment industry. So far a few 3D caricature generation methods exist and all of them require some caricature information (e.g., a caricature sketch or 2D caricature) as input. This kind of input, however, is difficult to provide by non-professional users. In this paper, we propose an end-to-end deep neural network model that generates high-quality 3D caricatures directly from a normal 2D face photo. The most challenging issue for our system is that the source domain of face photos (characterized by normal 2D faces) is significantly different from the target domain of 3D caricatures (characterized by 3D exaggerated face shapes and textures). To address this challenge, we: (1) build a large dataset of 5,343 3D caricature meshes and use it to establish a PCA model in the 3D caricature shape space; (2) reconstruct a normal full 3D head from the input face photo and use its PCA representation in the 3D caricature shape space to establish correspondences between the input photo and 3D caricature shape; and (3) propose a novel character loss and a novel caricature loss based on previous psychological studies on caricatures. Experiments including a novel two-level user study show that our system can generate high-quality 3D caricatures directly from normal face photos.
BibTeX
@article{ye2023d,
  author = {Zipeng Ye and Ran Yi and Minjing Yu and Juyong Zhang and Yu-Kun Lai and Yong-Jin Liu},
  title = {{3D-CariGAN: An End-to-End Solution to 3D Caricature Generation from Face Photos}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2023},
  doi = {10.1109/tvcg.2021.3126659},
}
Figure