Introduction
Facial Sketch Synthesis (FSS) is the computational task of generating artistic sketches from portrait photographs. This field has significant applications in digital entertainment and forensic face recognition. However, traditional approaches face three primary challenges: the scarcity of high-quality, professionally drawn sketch datasets; the rigidity of models typically limited to a single artistic style; and sensitivity to complex backgrounds or lighting conditions in input images.
Researchers have recently proposed a novel framework that shifts away from the often-unstable Generative Adversarial Networks (GANs) toward Masked Generative Modeling (MGM). This approach leverages transformer-based architectures and semi-supervised learning to produce high-quality, multi-style sketches from single 2D photographs without requiring auxiliary 3D geometry.
Methodology
The proposed system utilizes a Masked Generative Transformer to learn facial structures by predicting missing image components. The architecture consists of several key modules:
- VQ-GAN Tokenization: To ensure computational efficiency, $256 \times 256$ sketches are compressed into a $16 \times 16$ grid of discrete visual tokens using a VQ-GAN tokenizer.
- Masked Modeling: During training, a portion of these tokens is masked. The transformer is tasked with recovering the original tokens based on spatial context and photographic input.
- Feature Guidance: A pre-trained CLIP-L/14 encoder extracts deep semantic features from the input photo. These features guide the transformer through cross-attention and adaptive normalization, ensuring the output maintains high fidelity to the subject's identity.
- Two-Stage Training: The model is first pre-trained on the large-scale CelebA dataset using synthetic sketches to establish basic facial structural knowledge. It is then fine-tuned on the FS2K dataset, which contains 2,104 high-quality, artist-drawn photo-sketch pairs.
Multi-Style Synthesis and Interpolation
A significant innovation of this work is the treatment of "style" as a continuous parameter rather than a discrete label. By embedding style as a variable, the model can perform style interpolation, allowing users to generate sketches that blend characteristics of different artistic techniques.
Results and Performance
The MGM-based approach demonstrates superior performance compared to state-of-the-art (SOTA) models such as HIDA and FSGAN across several benchmarks:
- Fréchet Inception Distance (FID): The model achieved a score of 26.79, outperforming CycleGAN (29.13), indicating higher generation quality.
- Structural Similarity (SSIM): A score of 0.909 confirms high structural accuracy relative to the source photograph.
- Qualitative Quality: Visual evaluations indicate cleaner line work, more realistic hair textures, and a robust ability to isolate the subject from complex backgrounds.
Conclusion and Future Work
While the transition to masked generative modeling provides enhanced stability and quality, certain limitations remain. The model's reliance on a fixed VQ-GAN codebook may limit its ability to represent highly unique textures. Additionally, the iterative nature of transformer decoding can result in slower inference speeds compared to single-pass convolutional networks. Future research may focus on expanding the model's robustness to extreme facial poses and further optimizing decoding efficiency.
Summary updated