Disentangled Feature Networks for Facial Portrait and Caricature Generation

Kaihao Zhang, Wenhan Luo, Lin Ma, Wenqi Ren, Hongdong Li

Abstract

Generating high-quality facial caricatures requires a delicate balance between artistic exaggeration and the preservation of a subject's identity. This paper introduces a novel framework centered on a "four-stream" Disentangled Feature Network (DFN) that separates a face into its core components: geometry, texture, specific organs, and artistic style. By disentangling these features, the model can manipulate them independently to produce diverse, stylized portraits. Furthermore, an Adversarial Portrait Mapping Module (APMM) is proposed to automate the generation process by mapping random noise to specific geometric and stylistic latent spaces, allowing for the creation of varied caricatures from a single input image without manual intervention.

Introduction

Caricature generation is a complex task in computer vision due to the significant domain gap between natural photographs and artistic drawings. Unlike standard image-to-image translation, caricatures involve non-linear geometric warping alongside stylistic changes. Traditional methods often struggle with unpaired data and the complex interplay between facial features, where changing the shape of a feature (like a nose) often inadvertently degrades the skin texture or background style.

Conceptual overview of the disentanglement process for facial reconstruction and manipulation.

To address these challenges, we propose a framework that treats caricature generation as a disentanglement and reconstruction problem. By learning independent representations for geometry and style, the system gains fine-grained control over the final output, ensuring that identity-defining "anchors" remain intact even under heavy exaggeration.

Methods

Disentangled Feature Network (DFN)

The core of the proposed system is the DFN, which processes an input image through four distinct streams to capture essential facial information:

  1. Landmarks: This stream captures the overall geometric structure and expression, providing the framework for exaggeration.
  2. Face Texture: This component handles skin tone and general color information, separate from the underlying structure.
  3. Prominent Organs: By focusing specifically on the eyes, nose, and mouth, the network ensures these critical identity markers remain recognizable during the warping process.
  4. Artistic Style: This stream extracts stylistic information (e.g., brushstrokes, color palettes) from a reference portrait.
The detailed architecture of the Disentangled Feature Network (DFN), showing the multi-stream encoding and decoding process.

As shown in the architecture diagram, these features are encoded into a latent space and then recombined by a decoder to reconstruct the final caricature. This modular approach prevents the "ghosting" artifacts common in entangled models.

Adversarial Portrait Mapping Module (APMM)

To enable fully automatic generation, the APMM learns to map simple Gaussian noise into the latent spaces of styles and geometric exaggerations. This allows the system to "sample" different artistic choices, generating a wide variety of results from a single natural photo. This module effectively simulates the creative decisions an artist might make regarding the degree of exaggeration and the specific drawing style.

Datasets and Evaluation

The model was trained and evaluated using WebCaricature, the largest available caricature dataset (6,042 caricatures and 5,974 natural images), and CelebA, which provided a foundation for understanding general human facial geometry.

Qualitative comparisons against state-of-the-art methods like CycleGAN, CariGAN, and WarpGAN demonstrate that the DFN produces portraits with richer textures and more plausible exaggerations. Ablation studies confirmed that removing either the "Landmark" or "Organ" streams significantly degraded visual quality and identity preservation, proving that granular disentanglement is essential for this task.

Conclusion and Future Work

The Disentangled Feature Network offers a robust solution for automated caricature generation by separating identity, geometry, and style. While the system produces high-quality results, it currently faces a "Missing Hair Problem" due to standard face detection cropping, which often ignores complex hairstyles or the top of the head. Future research will focus on improving cropping robustness and exploring even more granular disentanglement, such as separating environmental lighting from intrinsic facial texture.

Summary updated

Abstract

Facial portrait is an artistic form which draws faces by emphasizing discriminative or prominent parts of faces via various kinds of drawing tools. However, the complex interplay between the different facial factors, such as facial parts, background, and drawing styles, and the significant domain gap between natural facial images and their portrait counterparts makes the task challenging. In this paper, a flexible four-stream Disentangled Feature Networks ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DFN</i> ) is proposed to learn disentangled feature representation of different facial factors and generate plausible portraits with reasonable exaggerations and richness in style. Four factors are encoded as embedding features, and combined to reconstruct facial portraits. Meanwhile, to make the process fully automatic (without manually specifying either portrait style or exaggerating form), we propose a new Adversarial Portrait Mapping Module ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">APMM</i> ) to map noise to the embedding feature space, as proxies for portrait style and exaggerating. Thanks to the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DFN</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">APMM</i> , we are able to manipulate the portrait style and facial geometric structures to generate a large number of portraits. Extensive experiments on two public datasets show that our proposed methods can generate a diverse set of artistic portraits.
BibTeX
@article{zhang2022disentangled,
  author = {Kaihao Zhang and Wenhan Luo and Lin Ma and Wenqi Ren and Hongdong Li},
  title = {{Disentangled Feature Networks for Facial Portrait and Caricature Generation}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
  doi = {10.1109/tmm.2021.3064273},
}
Figure