CariGANs: Unpaired Photo-to-Caricature Translation

Kaidi Cao, Jing Liao, Lu Yuan

TL;DR

This paper introduces CariGANs, the first Generative Adversarial Network (GAN) specifically designed for translating ordinary photos into caricatures without requiring paired examples. By decoupling the process into separate modules for geometric exaggeration and appearance stylization, CariGANs achieve more realistic and controllable caricature generation than previous methods.

What Problem Does This Paper Address?

Creating compelling caricatures traditionally requires skilled artists to both exaggerate facial features (geometry) and apply a distinct artistic style (appearance). Automating this process is challenging because:

  1. Unpaired Data: Real-world datasets of face photos and their corresponding caricatures are scarce and expensive to create, making supervised learning difficult.
  2. Dual Nature of Caricatures: A successful caricature involves two distinct transformations: exaggerating the underlying facial structure and applying a stylized visual appearance. Previous methods often struggled to handle both simultaneously, especially with unpaired data.
  3. Controllability: Users often want to control the degree of exaggeration or the specific artistic style, which is hard to achieve with existing automated systems.

The paper aims to bridge this gap by developing an unsupervised method that can generate high-quality, controllable caricatures from un-paired photo and caricature datasets.

Methods

CariGANs tackle the photo-to-caricature translation problem by breaking it down into two specialized Generative Adversarial Networks (GANs):

  • CariGeoGAN (Geometry Exaggeration): This module focuses solely on transforming the geometric structure of a face. Instead of working with raw images, it uses Principal Component Analysis (PCA) to represent facial landmarks as low-dimensional vectors. This representation helps preserve the overall face structure and makes the geometry translation more robust. CariGeoGAN employs a cycle-consistent GAN architecture (similar to CycleGAN) to learn the mapping from photo landmarks to exaggerated caricature landmarks and vice-versa. A novel "characteristic loss" is introduced to encourage the exaggeration of distinctive facial features while preserving their relative positions.
  • CariStyGAN (Appearance Stylization): This module handles the visual style transfer. To ensure it only learns appearance without being influenced by geometry changes, it uses an "intermediate domain" created by warping original caricatures to match the geometry of the input photos (using the output from CariGeoGAN). CariStyGAN then learns to transfer the appearance from these intermediate caricatures to the input photos. It also uses a cycle-consistent GAN framework but incorporates ideas from MUNIT to allow for multi-modal translation (generating diverse styles) and adds a perceptual loss (using VGG19 features) to better preserve the identity of the original face.

The final caricature is produced by warping the stylized photo (from CariStyGAN) using the exaggerated landmarks (from CariGeoGAN).

Datasets

  • Training Data:
    • Photos: 10,000 face images from the CelebA dataset.
  • Caricatures: 8,451 hand-drawn portrait caricatures collected from the internet, featuring diverse styles and exaggerations.
  • Evaluation Data:
    • The paper mentions using a separate testing dataset for evaluation, including images from CelebA not used in training.
  • For perceptual studies, a new dataset was constructed with 500 hand-drawn caricatures and 20 portrait photos per person to test identity preservation and style faithfulness.

Key Insights and Ideas

  • Decoupling Geometry and Appearance: The core innovation is recognizing that photo-to-caricature translation can be effectively split into two distinct, more manageable tasks: geometric exaggeration and appearance stylization. This decomposition simplifies the learning problem.
  • PCA for Landmark Representation: Using PCA on facial landmarks for geometry translation is crucial. It reduces dimensionality, enforces a face shape prior, and makes the geometry mapping more robust to small errors, preventing severe artifacts.
  • Intermediate Domain for Stylization: Creating an intermediate domain where geometry is aligned to the input photo but appearance is caricature-like is key to training CariStyGAN for pure appearance transfer without geometric distortions.
  • Controllability through Parameters: The separation of geometry and style allows for independent control. Users can adjust the exaggeration factor for geometry and select different styles through random sampling or reference images for appearance.
  • Characteristic Loss for Meaningful Exaggeration: The characteristic loss in CariGeoGAN is vital for ensuring that the geometric exaggeration focuses on distinctive features rather than arbitrary distortions, leading to more artistically relevant results.

Key Results

  • Perceptual Study: Caricatures generated by CariGANs were rated as closer to hand-drawn caricatures and better preserved identity compared to state-of-the-art methods like MUNIT and CycleGAN.
  • Identity Recognition: CariGANs achieved a high recognition rate, close to hand-drawn caricatures, and significantly better than MUNIT, Neural Style, and Deep Analogy.
  • Style Faithfulness: CariGANs ranked second only to hand-drawn caricatures in terms of visual style similarity, with users sometimes mistaking CariGAN results for real hand-drawn caricatures.
  • Performance: The system achieves a runtime of approximately 0.14 seconds per 256x256 image, with efficient breakdowns for exaggeration, stylization, and warping.
  • Controllability: The system demonstrated effective control over both geometric exaggeration (via a parameter $\alpha$) and appearance style (via random style codes or reference images).

Limitations and Future Work

  • Geometric Exaggeration Limitations: The current geometric exaggeration is more pronounced on the face contour due to the landmark representation. Minor exaggerations on features like ears or hair are not fully captured. The authors suggest adding more landmarks to address this.
  • Content-Style Disentanglement in Geometry: While appearance style is controllable, disentangling content and style in geometry is still unclear and not fully achieved.
  • Style Diversity: CariStyGAN performs well with common caricature styles (sketch, cartoon) but is less faithful to uncommon styles (e.g., oil painting) due to limited training data for style decoupling.
  • Resolution: The current system is trained on low-resolution images (256x256). Future work could explore progressive growing techniques for higher-resolution outputs.
  • Temporal Consistency for Videos: For video caricatures, temporal constraints could be added to reduce flickering between frames.
  • Multi-modal Geometry: Making the geometric exaggeration multi-modal, similar to the appearance stylization, is a potential future direction.

Summary updated

Abstract

Facial caricature is an art form of drawing faces in an exaggerated way to convey humor or sarcasm. In this paper, we propose the first Generative Adversarial Network (GAN) for unpaired photo-to-caricature translation, which we call "CariGANs". It explicitly models geometric exaggeration and appearance stylization using two components: CariGeoGAN, which only models the geometry-to-geometry transformation from face photos to caricatures, and CariStyGAN, which transfers the style appearance from caricatures to face photos without any geometry deformation. In this way, a difficult cross-domain translation problem is decoupled into two easier tasks. The perceptual study shows that caricatures generated by our CariGANs are closer to the hand-drawn ones, and at the same time better persevere the identity, compared to state-of-the-art methods. Moreover, our CariGANs allow users to control the shape exaggeration degree and change the color/texture style by tuning the parameters or giving an example caricature.
BibTeX
@article{cao2018carigans,
  author = {Kaidi Cao and Jing Liao and Lu Yuan},
  title = {{CariGANs: Unpaired Photo-to-Caricature Translation}},
  journal = {ACM Trans. Graphics},
  year = {2018},
  doi = {10.1145/3272127.3275046},
}
Figure