Learning to sketch human facial portraits using personal styles by case-based reasoning

Bingwen Jin, Songhua Xu, Weidong Geng

TL;DR

This paper introduces a novel Case-Based Reasoning (CBR) framework to automatically generate facial sketches from photographs while mimicking the distinct personal style of a specific human artist. By iteratively retrieving and adapting "cases" (photo-sketch pairs) using learned predictive models, the system captures both stroke-level stylization and geometric exaggeration, outperforming existing state-of-the-art methods using fewer than 50 training examples.

What Problem Does This Paper Address?

In the realm of Image-Based Artistic Rendering (IB-AR), generating sketches from photos usually relies on Example-Based Artistic Rendering (EBAR). Existing EBAR methods generally fall into two camps: model-based (which learn parametric rules but lose subtle, unique artistic quirks) and model-free (which reuse patches/pixels from examples but are limited by the exact data provided).

The core challenge is that human artists do not draw by simply applying a global filter or blindly copying patches. They draw progressively, step-by-step, and often apply personalized geometric exaggeration (caricature-like features) based on their unique style. Existing algorithms struggle to autonomously replicate this multi-step, highly personalized, and exaggerative process. This paper attempts to bridge that gap by teaching a system to emulate the specific stylistic and exaggerative habits of individual artists.

Methods

The authors frame sketch generation as a Case-Based Reasoning (CBR) problem. In CBR, a new problem is solved by retrieving similar past cases and adapting their solutions. Here, a "case" is a training sample consisting of a source photo, its corresponding hand-drawn sketch, a "neutralized" sketch (exaggeration removed), and extracted facial landmarks.

The pipeline works iteratively, region-by-region (based on 67 facial feature points), mimicking a human's progressive drawing process:

  1. Fitness Evaluation (FE) Model: Instead of just finding the most visually similar photo patch, the system uses an FE model to predict which candidate case will best help synthesize the final desired sketch (hypothesizing the "ground truth" in the artist's mind).
  2. Parameter Estimation (PE) Model: Once a case is retrieved, it must be adapted (blended) with the current interim sketch. The PE model automatically calculates the optimal blending weights to fuse the retrieved case with the draft.
  3. Exaggeration Field (EF): Because exaggeration is a hallmark of portrait sketching, the authors separate texture from geometry. They train a linear system to learn a 2D vector field representing how a specific artist warps facial features. After the iterative sketching is complete, this EF is applied as a post-processing step to warp the final image into the artist's signature exaggerated style.

Datasets

The authors utilized three datasets (denoted as $\Pi_1$, $\Pi_2$, and $\Pi_3$), each containing 49 front-view photos and corresponding sketches:

  • $\Pi_1$ (CUHK Student Dataset): A standard benchmark dataset. The sketches here are realistic and lack exaggeration.
  • $\Pi_2$ (Professional Artist): 49 photos (39 from CUHK, 10 new) sketched by an artist with >20 years of experience. These feature distinct personal exaggeration.
  • $\Pi_3$ (Freelance Illustrator): The same 49 photos sketched by a digital art PhD student, featuring a different, highly stylized caricature approach. (Note: $\Pi_2$ and $\Pi_3$ were commissioned specifically for this paper to test the model's ability to learn personalized exaggeration).

Key Insights and Ideas

  • Hypothesizing the "Mind's Eye": The most clever aspect of the paper is the Fitness Evaluation model. By training a model to evaluate cases based on how close they get the interim sketch to a hypothesized "ground truth," the algorithm effectively simulates an artist looking at their canvas, imagining the final result, and deciding what stroke to make next.
  • Decoupling Style and Geometry: By neutralizing the training sketches (aligning their geometry back to the source photo) before learning the stroke styles, the system avoids confusing texture/shading with structural exaggeration. Handling exaggeration via a separate 2D vector field makes the pipeline highly modular.
  • High Data Efficiency: Because the CBR approach intelligently adapts existing cases rather than requiring massive datasets to learn generalized latent spaces, the system successfully captures an artist's style using fewer than 50 examples.

Key Results

  • Quantitative (User Study): In a blind online survey with 2,624 participants (yielding 26,240 answers), 62% of respondents preferred the sketches generated by this CBR method over those generated by a state-of-the-art peer algorithm (Wang et al., 2009).
  • Efficiency: The method requires only 48 training samples to achieve these results, compared to the 88+ samples required by competing methods.
  • Speed: Generating a new sketch takes about 1 minute on a standard CPU (unoptimized), compared to 3 minutes for the peer algorithm.
  • Qualitative: Visual comparisons show the CBR method successfully preserves crucial stylistic details (like nasolabial folds) and accurately applies the distinct geometric exaggerations of Artists II and III, which peer methods and commercial software (Akvis Sketch) completely fail to capture.

Limitations and Future Work

The authors explicitly acknowledge one major limitation: the system cannot synthesize human hair. The algorithm relies heavily on the 67 anatomical facial landmarks to segment regions and establish correspondences between the photo and the sketch. Because hair lacks consistent, definable anatomical landmarks across different people, the system cannot currently map hair textures or strokes.

Future work suggests exploring Cross-Modal Face Matching or allowing users to manually indicate keypoints on the hair to guide the synthesis process. Additionally, the authors note that parallelizing the independent regional models could reduce the current 8-hour training time significantly.

Summary updated

Abstract

This paper employs case-based reasoning (CBR) to capture the personal styles of individual artists and generate the human facial portraits from photos accordingly. For each human artist to be mimicked, a series of cases are firstly built-up from her/his exemplars of source facial photo and hand-drawn sketch, and then its stylization for facial photo is transformed as a style-transferring process of iterative refinement by looking-for and applying best-fit cases in a sense of style optimization. Two models, fitness evaluation model and parameter estimation model, are learned for case retrieval and adaptation respectively from these cases. The fitness evaluation model is to decide which case is best-fitted to the sketching of current interest, and the parameter estimation model is to automate case adaptation. The resultant sketch is synthesized progressively with an iterative loop of retrieval and adaptation of candidate cases until the desired aesthetic style is achieved. To explore the effectiveness and advantages of the novel approach, we experimentally compare the sketch portraits generated by the proposed method with that of a state-of-the-art example-based facial sketch generation algorithm as well as a couple commercial software packages. The comparisons reveal that our CBR based synthesis method for facial portraits is superior both in capturing and reproducing artists’ personal illustration styles to the peer methods.
BibTeX
@article{jin2018learning,
  author = {Bingwen Jin and Songhua Xu and Weidong Geng},
  title = {{Learning to sketch human facial portraits using personal styles by case-based reasoning}},
  journal = {Multimedia Tools and Applications},
  year = {2018},
  doi = {10.1007/s11042-017-4457-8},
}

Loading related content...

Figure