TL;DR
This paper introduces a novel Case-Based Reasoning (CBR) framework to automatically generate facial sketches from photographs while mimicking the distinct personal style of a specific human artist. By iteratively retrieving and adapting "cases" (photo-sketch pairs) using learned predictive models, the system captures both stroke-level stylization and geometric exaggeration, outperforming existing state-of-the-art methods using fewer than 50 training examples.
What Problem Does This Paper Address?
In the realm of Image-Based Artistic Rendering (IB-AR), generating sketches from photos usually relies on Example-Based Artistic Rendering (EBAR). Existing EBAR methods generally fall into two camps: model-based (which learn parametric rules but lose subtle, unique artistic quirks) and model-free (which reuse patches/pixels from examples but are limited by the exact data provided).
The core challenge is that human artists do not draw by simply applying a global filter or blindly copying patches. They draw progressively, step-by-step, and often apply personalized geometric exaggeration (caricature-like features) based on their unique style. Existing algorithms struggle to autonomously replicate this multi-step, highly personalized, and exaggerative process. This paper attempts to bridge that gap by teaching a system to emulate the specific stylistic and exaggerative habits of individual artists.
Methods
The authors frame sketch generation as a Case-Based Reasoning (CBR) problem. In CBR, a new problem is solved by retrieving similar past cases and adapting their solutions. Here, a "case" is a training sample consisting of a source photo, its corresponding hand-drawn sketch, a "neutralized" sketch (exaggeration removed), and extracted facial landmarks.
The pipeline works iteratively, region-by-region (based on 67 facial feature points), mimicking a human's progressive drawing process:
- Fitness Evaluation (FE) Model: Instead of just finding the most visually similar photo patch, the system uses an FE model to predict which candidate case will best help synthesize the final desired sketch (hypothesizing the "ground truth" in the artist's mind).
- Parameter Estimation (PE) Model: Once a case is retrieved, it must be adapted (blended) with the current interim sketch. The PE model automatically calculates the optimal blending weights to fuse the retrieved case with the draft.
- Exaggeration Field (EF): Because exaggeration is a hallmark of portrait sketching, the authors separate texture from geometry. They train a linear system to learn a 2D vector field representing how a specific artist warps facial features. After the iterative sketching is complete, this EF is applied as a post-processing step to warp the final image into the artist's signature exaggerated style.
Datasets
The authors utilized three datasets (denoted as $\Pi_1$, $\Pi_2$, and $\Pi_3$), each containing 49 front-view photos and corresponding sketches:
- $\Pi_1$ (CUHK Student Dataset): A standard benchmark dataset. The sketches here are realistic and lack exaggeration.
- $\Pi_2$ (Professional Artist): 49 photos (39 from CUHK, 10 new) sketched by an artist with >20 years of experience. These feature distinct personal exaggeration.
- $\Pi_3$ (Freelance Illustrator): The same 49 photos sketched by a digital art PhD student, featuring a different, highly stylized caricature approach. (Note: $\Pi_2$ and $\Pi_3$ were commissioned specifically for this paper to test the model's ability to learn personalized exaggeration).
Key Insights and Ideas
- Hypothesizing the "Mind's Eye": The most clever aspect of the paper is the Fitness Evaluation model. By training a model to evaluate cases based on how close they get the interim sketch to a hypothesized "ground truth," the algorithm effectively simulates an artist looking at their canvas, imagining the final result, and deciding what stroke to make next.
- Decoupling Style and Geometry: By neutralizing the training sketches (aligning their geometry back to the source photo) before learning the stroke styles, the system avoids confusing texture/shading with structural exaggeration. Handling exaggeration via a separate 2D vector field makes the pipeline highly modular.
- High Data Efficiency: Because the CBR approach intelligently adapts existing cases rather than requiring massive datasets to learn generalized latent spaces, the system successfully captures an artist's style using fewer than 50 examples.
Key Results
- Quantitative (User Study): In a blind online survey with 2,624 participants (yielding 26,240 answers), 62% of respondents preferred the sketches generated by this CBR method over those generated by a state-of-the-art peer algorithm (Wang et al., 2009).
- Efficiency: The method requires only 48 training samples to achieve these results, compared to the 88+ samples required by competing methods.
- Speed: Generating a new sketch takes about 1 minute on a standard CPU (unoptimized), compared to 3 minutes for the peer algorithm.
- Qualitative: Visual comparisons show the CBR method successfully preserves crucial stylistic details (like nasolabial folds) and accurately applies the distinct geometric exaggerations of Artists II and III, which peer methods and commercial software (Akvis Sketch) completely fail to capture.
Limitations and Future Work
The authors explicitly acknowledge one major limitation: the system cannot synthesize human hair. The algorithm relies heavily on the 67 anatomical facial landmarks to segment regions and establish correspondences between the photo and the sketch. Because hair lacks consistent, definable anatomical landmarks across different people, the system cannot currently map hair textures or strokes.
Future work suggests exploring Cross-Modal Face Matching or allowing users to manually indicate keypoints on the hair to guide the synthesis process. Additionally, the authors note that parallelizing the independent regional models could reduce the current 8-hour training time significantly.
Summary updated