Learning to Sketch Human Facial Portraits using Personal Styles by Case-Based Reasoning

Bingwen Jin, Songhua Xu, Weidong Geng

Abstract

This paper introduces a novel framework employing case-based reasoning (CBR) to capture the personal styles of individual artists and automatically generate facial sketches from photographs. By iteratively retrieving and adapting "cases"—pairs of source facial photos and hand-drawn sketches—the system transforms stylization into a process of iterative refinement. Two learned models, a fitness evaluation model for case retrieval and a parameter estimation model for case adaptation, guide the synthesis. Experimental results demonstrate that this CBR-based method effectively reproduces both stroke-level stylization and geometric exaggeration, outperforming state-of-the-art example-based algorithms and commercial software.

Introduction

Generating artistic sketches from photographs, a subfield of Image-Based Artistic Rendering (IB-AR), typically relies on Example-Based Artistic Rendering (EBAR). Traditional EBAR methods are often either model-based, which use parametric rules that may lose subtle artistic nuances, or model-free, which reuse patches from examples but are limited by the specific data provided.

Human artists do not draw by applying global filters or blindly copying patches; they work progressively and often incorporate personalized geometric exaggerations. Existing algorithms struggle to replicate this multi-step, highly personalized process. This research addresses this gap by teaching a system to emulate the specific stylistic and exaggerative habits of individual artists through a Case-Based Reasoning framework.

Methodology

The proposed approach frames sketch generation as a CBR problem, where new sketches are synthesized by retrieving and adapting similar past cases. A "case" consists of a source photo, its corresponding hand-drawn sketch, a "neutralized" sketch (with exaggeration removed), and extracted facial landmarks.

The pipeline operates iteratively region-by-region, based on 67 facial feature points:

  1. Fitness Evaluation (FE) Model: This model predicts which candidate case will best help synthesize the final desired sketch by hypothesizing the "ground truth" in the artist's mind.
  2. Parameter Estimation (PE) Model: This model automatically calculates optimal blending weights to fuse the retrieved case with the current interim sketch.
  3. Exaggeration Field (EF): To handle caricature-like features, the authors decouple texture from geometry. A linear system learns a 2D vector field representing an artist's specific warping style, which is applied as a post-processing step to the final sketch.

Experimental Setup

The model was evaluated using three datasets, each containing 49 photo-sketch pairs:

  • $Π_1$ (CUHK Student Dataset): A standard benchmark with realistic sketches.
  • $Π_2$ (Professional Artist): Sketches by an artist with over 20 years of experience, featuring distinct personal exaggeration.
  • $Π_3$ (Freelance Illustrator): Sketches by a digital art PhD student, featuring a highly stylized caricature approach.

Results and Discussion

The CBR method demonstrates high data efficiency, successfully capturing an artist's style with fewer than 50 examples. By decoupling style and geometry, the system avoids confusing texture with structural exaggeration.

Comparison of the proposed CBR method against peer methods and human artist originals.

A large-scale user study involving over 2,600 participants revealed that 62% of respondents preferred the sketches generated by this CBR method over those from state-of-the-art peer algorithms. Qualitatively, the method preserves crucial stylistic details, such as nasolabial folds, and accurately reproduces the geometric exaggerations characteristic of specific artists, which competing methods and commercial software often fail to capture. Furthermore, the algorithm is computationally efficient, generating a sketch in approximately one minute on a standard CPU.

Conclusion and Future Work

While the system excels at facial features, it currently cannot synthesize human hair due to the lack of consistent anatomical landmarks. Future research may explore cross-modal face matching or manual keypoint indication to address this. Additionally, parallelizing the regional models could significantly reduce training time, which currently stands at approximately eight hours. Overall, the CBR framework provides a robust and efficient means of mimicking personalized artistic styles in facial portraiture.

Summary updated

Abstract

This paper employs case-based reasoning (CBR) to capture the personal styles of individual artists and generate the human facial portraits from photos accordingly. For each human artist to be mimicked, a series of cases are firstly built-up from her/his exemplars of source facial photo and hand-drawn sketch, and then its stylization for facial photo is transformed as a style-transferring process of iterative refinement by looking-for and applying best-fit cases in a sense of style optimization. Two models, fitness evaluation model and parameter estimation model, are learned for case retrieval and adaptation respectively from these cases. The fitness evaluation model is to decide which case is best-fitted to the sketching of current interest, and the parameter estimation model is to automate case adaptation. The resultant sketch is synthesized progressively with an iterative loop of retrieval and adaptation of candidate cases until the desired aesthetic style is achieved. To explore the effectiveness and advantages of the novel approach, we experimentally compare the sketch portraits generated by the proposed method with that of a state-of-the-art example-based facial sketch generation algorithm as well as a couple commercial software packages. The comparisons reveal that our CBR based synthesis method for facial portraits is superior both in capturing and reproducing artists' personal illustration styles to the peer methods.
BibTeX
@article{jin2018learning,
  author = {Bingwen Jin and Songhua Xu and Weidong Geng},
  title = {{Learning to Sketch Human Facial Portraits using Personal Styles by Case-Based Reasoning}},
  journal = {Multimedia Tools and Applications},
  year = {2018},
  doi = {10.1007/s11042-017-4457-8},
}
Figure