Lighting and Pose Robust Face Sketch Synthesis

Wei Zhang, Xiaogang Wang, Xiaoou TangECCV 2010 · DOI

TL;DR

This paper introduces a novel algorithm for generating realistic face sketches from photographs, even when the photos have varying lighting and poses compared to the training data. By incorporating shape priors, robust patch matching, and improved compatibility terms within a multiscale Markov Random Field (MRF) framework, the method significantly outperforms existing state-of-the-art techniques in producing accurate and artifact-free sketches.

What Problem Does This Paper Address?

Automatic face sketch synthesis has numerous applications, from aiding law enforcement in identifying suspects to enhancing digital entertainment. While previous methods have made strides, they often falter when presented with input photos that differ in lighting or pose from the training examples. This limitation restricts their practical use in real-world scenarios where such variations are common. The paper aims to bridge this gap by developing a robust synthesis system that can handle these challenging variations, producing high-quality sketches from diverse input images.

Methods

The core of the proposed method is a multiscale Markov Random Field (MRF) model. To achieve robustness to lighting and pose variations, the authors introduce several key innovations:

  • Shape Priors: Specific shape priors for facial components are integrated into the MRF model. These priors help to reduce distortions and artifacts caused by lighting and pose changes, and crucially, restore lost facial structures.
  • Robust Patch Descriptors and Metrics: The paper employs patch descriptors and metrics that are more resilient to lighting variations. This includes a "local evidence" function that combines both photo-to-photo patch matching (similar to previous methods) and photo-to-sketch patch matching. This dual approach improves matching accuracy, especially under challenging lighting.
  • Difference-of-Gaussians (DoG) Filtering: Before patch matching, input photos are processed with a DoG filter. This enhances edges for better facial detail and, by subtracting low-frequency components, reduces the impact of shading effects and lighting variations.
  • Improved Neighboring Compatibility: A smoothing term is introduced that considers not only intensity compatibility but also gradient compatibility between neighboring sketch patches. This helps to ensure smoother transitions and reduce artifacts.

Datasets

The authors evaluate their method on two datasets:

  • CUHK Face Sketch Database: This dataset is commonly used in face sketch synthesis research and contains face photos and corresponding artist-drawn sketches. For training, 88 persons with frontal photos under normal lighting were used. For testing, three subsets were created: a baseline set (frontal, normal lighting), a lighting variation set (frontal, three different lighting conditions), and a pose variation set (left/right poses, 45 degrees, normal lighting).
  • Celebrity Face Photos: A set of celebrity face photos from the web with uncontrolled lighting, poses, and backgrounds was also used to test the method's robustness in more challenging, real-world conditions.

Key Insights and Ideas

  • The Importance of Shape Priors: The paper highlights that simply relying on patch matching is insufficient when dealing with pose and lighting variations. Explicitly incorporating shape priors for facial components is crucial for restoring lost structures and preventing artifacts.
  • Dual Patch Matching is Key: Combining both photo-to-photo and photo-to-sketch patch matching significantly improves robustness. This acknowledges that while photos and sketches are different modalities, there are underlying structural similarities that can be leveraged.
  • DoG Filtering for Preprocessing: The DoG filter proves to be an effective preprocessing step, enhancing facial details and normalizing lighting variations better than simple luminance remapping in this context.
  • Gradient Compatibility Matters: Adding gradient compatibility to the neighboring term in the MRF helps to preserve structural integrity, preventing the loss of features that can occur when only intensity smoothing is applied.

Key Results

  • Significant Performance Improvement: The proposed method demonstrates a substantial improvement over the state-of-the-art method [5] in synthesizing sketches from photos with lighting and pose variations.
  • Quantitative Evaluation: On the CUHK database, the method achieves higher rank-1 and rank-10 recognition rates when using synthesized sketches for matching against artist-drawn sketches, particularly under challenging lighting conditions (e.g., dark side lighting, achieving 71% rank-1 vs. 25.5% for the baseline method).
  • Qualitative Improvements: Visual results show that the proposed method produces sketches with fewer distortions, better facial details, and restored structures, especially in challenging cases where previous methods fail (e.g., hair, profile regions).
  • Efficiency: The method takes approximately 2 minutes to synthesize a sketch on a computer with a 3.20 GHz CPU, which is considered reasonable for practical applications.

Limitations and Future Work

  • Limitations: The paper notes that while CLAHE preprocessing can help with side lighting, it sometimes deteriorates performance, and the authors opt to report results without it in such cases. The CUHK database is also noted as not being suitable for training sketches of people from other ethnic groups due to human perception differences.
  • Future Work: The authors express interest in further investigating face sketch synthesis with expression variations, suggesting this as a natural extension of their robust framework.

Summary updated

BibTeX
@inproceedings{zhang2010lighting,
  author = {Wei Zhang and Xiaogang Wang and Xiaoou Tang},
  title = {{Lighting and Pose Robust Face Sketch Synthesis}},
  booktitle = {European Conference on Computer Vision},
  year = {2010},
  doi = {10.1007/978-3-642-15567-3\_31},
}
Figure