Abstract
This research presents a novel Generative Adversarial Network (GAN) architecture designed to synthesize realistic photographic faces from sparse sketches guided by specific attribute vectors. By framing the task as a "face hallucination" problem, the proposed model utilizes a dual-branch feature extraction strategy to combine structural constraints from sketches with semantic information from attributes like hair color or makeup. Experimental results on the CelebA dataset demonstrate that this approach achieves superior performance in terms of structural similarity and visual fidelity compared to existing state-of-the-art methods.
Introduction
The generation of realistic facial images from sketches is a fundamental challenge in computer vision, often characterized as an under-determined problem. While a sketch provides essential structural information—such as the positioning of facial features—it lacks the necessary details regarding texture, skin tone, and color. Previous methodologies have typically fallen into two categories: sketch-only methods, which often produce desaturated or blurry results, and text-to-image methods, which offer realistic textures but lack precise structural control.
This paper introduces a framework that bridges this gap by fusing sketch inputs with attribute descriptions. This combination ensures that the generated output is both structurally faithful to the input sketch and semantically consistent with desired characteristics.
Methodology
The core of the proposed system is a GAN framework featuring several architectural innovations:
Dual-Branch Feature Extraction
Rather than simple concatenation, the generator employs two distinct sub-networks. Branch A is dedicated to processing the sketch image to extract low-level profile and structural information. Branch B processes an 18-dimensional attribute vector, which is expanded and reshaped into a feature map to capture high-level semantic features such as color and texture.
Perceptual Fusion and Skip-Connections
The features from both branches are fused early in the network, allowing the model to learn the spatial relationship between attributes and structures (e.g., mapping "lipstick" to the lip boundaries). To preserve fine-grained spatial details, skip-connections are implemented between downsampling and upsampling layers, similar to the U-Net architecture.
Objective Function
The model is optimized using a multi-part loss function:
- Reid-wise Reconstruction Loss: Ensures pixel-level proximity to the ground truth.
- Adversarial Loss: Encourages the generation of images indistinguishable from real photographs.
Experimental Setup
The model was trained and evaluated using the Celebrity Face Attributes (CelebA) dataset.
- Data Preparation: 100,000 images were used for training and 10,000 for testing. Sketches were synthesized using the XDoG (Extended Difference of Gaussians) method.
- Attribute Selection: 18 key attributes (e.g., gender, hair color, beard type) were selected based on their significant visual impact.
Results and Discussion
The proposed model was benchmarked against Pix2Pix and contextual GANs. Quantitatively, the model achieved a PSNR of 16.3069 and an SSIM of 0.5790, outperforming baseline methods.
Qualitatively, the model demonstrates exceptional ability in generating localized details. For instance, the application of "lipstick" is precisely constrained within the lip boundaries defined by the sketch. The model also handles complex occlusions like sunglasses more naturally than previous approaches.
Conclusion
By treating sketch-to-photo translation as a face hallucination task and employing a dual-branch fusion strategy, this research provides a robust method for controllable face generation. Future work will focus on refining the feature extraction process to handle a broader range of attributes and improving the naturalness of transitions for complex facial accessories.
Summary updated