Attribute-controlled face photo synthesis from simple line drawing

Qi Guo, Ce Zhu, Zhiqiang Xia, Zhengtao Wang, Yipeng Liu

TL;DR

Researchers have developed a new deep learning model, the Attribute-Disentangled Variational Auto-Encoder (AD-VAE), that transforms simple, sparse line drawings into photorealistic face images. Unlike previous methods, this system allows users to "dial in" specific traits—like hair color, age, or complexion—giving them fine-grained control over the final synthesized photo.

What Problem Does This Paper Address?

Synthesizing a realistic photo from a simple line drawing is a "one-to-many" problem. A basic contour of a face doesn't contain information about skin tone, eye color, or lighting; therefore, infinitely many photos could technically match a single sketch.

While previous "exemplar-based" methods tried to map sketches to photos using large databases, they struggled to generalize to new faces or complex backgrounds. Recent deep learning approaches improved generalization but often lacked "controllability"—the user couldn't easily specify that the person in the sketch should have blond hair or be wearing glasses. This paper aims to bridge that gap by creating a model that understands and separates (disentangles) specific facial attributes from the general "style" of the image.

Methods

The authors introduce the Attribute-Disentangled Variational Auto-Encoder (AD-VAE). Here is the breakdown of the technical approach:

  • Latent Space Splitting: Standard VAEs compress an image into a single "latent" vector. AD-VAE splits this vector into two parts: $z_y$ (representing specific attributes like "male," "young," or "bangs") and $z_o$ (representing everything else, like lighting, pose, and background).
  • Supervised Disentanglement: By using binary attribute labels during training, the model learns to force specific dimensions of the latent space to correspond to specific physical traits.
  • Sketch Integration: To handle line drawings, the authors add a dedicated convolutional neural network channel. The features extracted from the sketch are concatenated with the latent vectors ($z_y$ and $z_o$) and fed into the decoder to generate the final photo.
  • Regularization: They use specific coefficients ($\alpha$ and $\beta$) in their loss function to balance the trade-off between how accurately the model classifies attributes and how well it reconstructs the overall image.

Datasets

The study utilizes two primary datasets:

  1. CelebA: A massive dataset of over 200,000 celebrity images with 40 attribute annotations. To create training data, the authors used a "Flow-based Difference-of-Gaussian" (FDoG) filter to automatically generate synthetic line drawings from these photos.
  2. ZJU-VIPA Line Drawing Face Database: Used for further testing, this dataset provides more traditional sketch-style data.

Key Insights and Ideas

  • Attributes as Content, Style as Context: The authors treat the line drawing and specific attributes as the "content" of the face, while regarding background and illumination as the "style." This allows them to perform style transfer, where they take the facial structure from a sketch but apply the lighting and background from a completely different "style photo."
  • Disentanglement vs. Conditioning: Unlike standard conditional VAEs (AC-VAE), which might "hallucinate" details to fill gaps (like adding teeth to a closed mouth just because the "smiling" attribute is on), AD-VAE's disentangled approach leads to more natural transitions and fewer visual artifacts.
  • Handling Incomplete Data: The model is robust enough to generate a convincing face even when the input line drawing is missing parts of the contour, relying on the learned latent representations to fill in the blanks.

Key Results

  • Qualitative Superiority: Compared to existing models like "Scribbler" or "Convolutional Sketch Inversion" (CSI), AD-VAE produces images that look more photorealistic and adhere more strictly to the requested attributes.
  • Attribute Manipulation: The paper demonstrates that by smoothly varying a single attribute variable (e.g., increasing the "mouth open" value), the generated image changes realistically without distorting other features like the eyes or hair.
  • Style Swapping: The model successfully synthesized photos using the geometry of a sketch but the environmental "style" (lighting/color palette) of a target reference photo.

Limitations and Future Work

  • Resolution: The experiments were conducted at a relatively low resolution ($64 \times 64$ pixels). Scaling this to high-definition photorealism remains a challenge for future iterations.
  • Attribute Scope: While the model handles 38 attributes, it is still limited to the labels provided in the training set (CelebA). It cannot currently control traits it hasn't been explicitly trained on.
  • Downstream Applications: The authors suggest that this "target style synthesis" could eventually be used to improve sketch-based face recognition and verification in law enforcement contexts by normalizing lighting and background across different samples.

Summary updated

Abstract

Face photo synthesis from simple line drawing is a one-to-many task as simple line drawing merely contains the contour of human face. Previous exemplar-based methods are over-dependent on the datasets and are hard to generalize to complicated natural scenes. Recently, several works utilize deep neural networks to increase the generalization, but they are still limited in the controllability of the users. In this paper, we propose a deep generative model to synthesize face photo from simple line drawing controlled by face attributes such as hair color and complexion. In order to maximize the controllability of face attributes, an attribute-disentangled variational auto-encoder (AD-VAE) is firstly introduced to learn latent representations disentangled with respect to specified attributes. Then we conduct photo synthesis from simple line drawing based on AD-VAE. Experiments show that our model can well disentangle the variations of attributes from other variations of face photos and synthesize detailed photorealistic face images with desired attributes. Regarding background and illumination as the style and human face as the content, we can also synthesize face photos with the target style of a style photo.
BibTeX
@inproceedings{guo2017attribute,
  author = {Qi Guo and Ce Zhu and Zhiqiang Xia and Zhengtao Wang and Yipeng Liu},
  title = {{Attribute-controlled face photo synthesis from simple line drawing}},
  booktitle = {ICIP},
  year = {2017},
  url = {https://openreview.net/forum?id=90Uldm8EFcJ},
}

Loading related content...

Figure