Abstract
Generating photorealistic face images from sparse, hand-drawn sketches is a challenging image-to-image translation task. Traditional convolutional neural networks often struggle with long-range dependencies, leading to structural inconsistencies like asymmetric features. This paper introduces CSAGAN (Conditional Self-Attention Generative Adversarial Network), a model designed to maintain global structural harmony and robustness to incomplete sketches. By combining a conditional self-attention mechanism with a multi-scale discriminator and a Dense Distance Field (DDF) input representation, CSAGAN achieves state-of-the-art results in face synthesis.
Introduction
The task of "image-to-image translation" has seen significant progress with models like pix2pix. However, these models rely heavily on local convolutional operations, which limit their ability to capture global context. In face generation, this often results in misaligned features or a failure to handle sparse, incomplete sketches. For instance, a model might fail to maintain symmetry between eyes or struggle to "fill in" missing facial components if they are omitted from the initial drawing. CSAGAN addresses these issues by incorporating global attention mechanisms and robust feature extraction.
Methodology
The CSAGAN architecture introduces several key innovations to improve the quality and structural integrity of generated faces:
1. Dense Distance Field (DDF)
Instead of using raw binary line maps, the authors convert sketches into an unsigned Euclidean distance field. This approach spreads the structural information across the entire image, providing a richer signal for the network and making it more resilient to gaps or noise in the input sketch.
2. Conditional Self-Attention Module (CSAM)
The CSAM serves as the core of the generator. By concatenating feature maps with the original input line map, the module allows the network to calculate relationships between distant pixels while remaining anchored to the user's intended structure. This ensures that features like the jawline and forehead are globally consistent.
3. Multi-Scale Discriminator and Feature Matching Loss
To balance global completeness with fine-grained detail, the model employs a multi-scale discriminator. A large-scale discriminator evaluates the overall facial structure, while a small-scale one focuses on textures like skin and hair. Furthermore, the authors use a feature matching loss based on the internal features of their face-trained discriminator to achieve more realistic textures.
Experimental Results
The model was trained on the CelebA-HD dataset, using 24,000 images for training and 6,000 for testing. Ground truth sketches were generated using a thinned HED edge detector to simulate human drawing styles.
Quantitative and Qualitative Performance
CSAGAN significantly outperformed baseline models such as pix2pix and SketchyGAN. Quantitatively, CSAGAN achieved a Fréchet Inception Distance (FID) of 269.59, a substantial improvement over pix2pix's 605.97. Qualitatively, the model produced sharper hair textures and more realistic skin illumination.
In a blind user study, participants preferred CSAGAN's results 93.1% of the time. Ablation studies further confirmed that the CSAM is critical; removing it led to a dramatic decrease in performance and structural coherence.
Conclusion and Future Work
CSAGAN demonstrates that integrating self-attention into GAN architectures can solve the long-range dependency problem in face synthesis. While highly effective on the CelebA-HD dataset, future research could explore the model's performance on extreme poses, complex occlusions, and more diverse artistic sketching styles. Additionally, optimizing the computational cost of self-attention modules remains a priority for potential real-time applications.
Summary updated