Abstract
Face photo synthesis from simple line drawing is a one-to-many task as simple line drawing merely contains the contour of human face. Previous exemplar-based methods are over-dependent on the datasets and are hard to generalize to complicated natural scenes. Recently, several works utilize deep neural networks to increase the generalization, but they are still limited in the controllability of the users. In this paper, we propose a deep generative model to synthesize face photo from simple line drawing controlled by face attributes such as hair color and complexion. In order to maximize the controllability of face attributes, an attribute-disentangled variational auto-encoder (AD-VAE) is firstly introduced to learn latent representations disentangled with respect to specified attributes. Then we conduct photo synthesis from simple line drawing based on AD-VAE. Experiments show that our model can well disentangle the variations of attributes from other variations of face photos and synthesize detailed photorealistic face images with desired attributes. Regarding background and illumination as the style and human face as the content, we can also synthesize face photos with the target style of a style photo.
Introduction
The task of generating a realistic photograph from a sparse line drawing is inherently under-constrained. A simple contour provides structural information but lacks critical details such as skin tone, eye color, or environmental lighting. This "one-to-many" mapping problem has historically been addressed with exemplar-based methods that rely heavily on large databases. However, these methods often fail to generalize to novel faces or complex backgrounds. While modern deep learning approaches have improved generalization, they frequently lack user controllability—the ability to specify particular traits like hair color or age. This research introduces the Attribute-Disentangled Variational Auto-Encoder (AD-VAE) to bridge this gap, offering fine-grained control over synthesized facial features.
Methodology
The core of the proposed system is the AD-VAE, which aims to disentangle facial attributes from the overall image style. Unlike standard VAEs that compress an image into a single latent vector, the AD-VAE splits the latent space into two distinct components: $z_y$, representing specific attributes (e.g., gender, age, bangs), and $z_o$, capturing remaining variations such as lighting, pose, and background.
To achieve this disentanglement, the model uses supervised learning with binary attribute labels during training. This forces specific dimensions of the latent space to correspond to physical traits. For sketch-to-photo synthesis, a dedicated convolutional neural network channel extracts features from the line drawing. These features are then concatenated with the latent vectors ($z_y$ and $z_o$) and processed by a decoder to generate the final photorealistic image. A balanced loss function, regulated by coefficients $\alpha$ and $\beta$, ensures a trade-off between attribute classification accuracy and overall reconstruction quality.
Results and Discussion
The AD-VAE demonstrates significant qualitative improvements over baseline models such as "Scribbler" and "Convolutional Sketch Inversion." The model produces images that are not only more photorealistic but also more responsive to attribute modifications. By smoothly varying values in the $z_y$ latent space, users can manipulate specific traits—such as hair color or skin tone—without affecting unrelated features like facial geometry or background lighting.
Furthermore, the model enables "style transfer" by treating the line drawing as content and the environmental factors (lighting, color palette) as style. By combining the structural features of a sketch with the latent style vector $z_o$ from a reference photo, the system can synthesize a face that matches the sketch's identity but adopts the reference photo's aesthetic context. Unlike standard conditional VAEs (AC-VAE), which might "hallucinate" details to fill gaps, AD-VAE's disentangled approach leads to more natural transitions and fewer visual artifacts.
Conclusion
The AD-VAE provides a robust framework for attribute-controlled face synthesis. While currently limited by a $64 \times 64$ pixel resolution and the scope of training labels in the CelebA dataset, the model shows great promise. Future work will focus on scaling to higher resolutions and expanding the range of controllable attributes. Potential applications include law enforcement, where target style synthesis could normalize lighting and background in sketches to improve face recognition and verification accuracy.
Summary updated