Caricature Synthesis Based on Mean Value Coordinates

Hongchuan Yu, Jianjun ZhangBournemouth 2010

Abstract

In this paper, a novel method for caricature synthesis is developed based on mean value coordinates (MVC). Our method can be applied to any single frontal face image to learn a specified caricature face exemplar pair for frontal and side view caricature synthesis. The technique only requires one or a small number of caricature face pairs and a natural frontal face training set, while the system can transfer the style of the exemplar pair across individuals. Further exaggeration can be fulfilled in a controllable way. Our method is further extended to facial expression transfer, interpolation and exaggeration, which are applications of expression editing. Moreover, the deformation equation of MVC is modified to handle the case of polygon intersections and applied to lateral view caricature synthesis from a single frontal view image. Using experiments we demonstrate that the transferred expressions are credible and the resulting caricatures can be characterized and recognized.

Introduction

Creating a high-quality caricature requires a delicate balance: the artist must exaggerate a subject's features for stylistic effect while ensuring the subject remains instantly recognizable. Automating this process is challenging because existing methods often require large datasets to learn styles, struggle to maintain identity after extreme deformation, or are strictly limited to frontal views. Generating a lateral (side) view caricature from a single frontal image is particularly difficult because turning a 2D face introduces 3D occlusions, causing the 2D control polygons used in image deformation to intersect and break standard algorithms. This paper introduces a novel, highly controllable method for generating facial caricatures and transferring expressions using Mean Value Coordinates (MVC).

Methodology

The core engine of this research is Mean Value Coordinates (MVC), a mathematical technique used in computer graphics to smoothly deform an image based on the movement of a surrounding control polygon. The authors decouple the caricature process into two distinct stages:

  • Shape Exaggeration: This alters the actual contours of facial components (e.g., making a nose pointier). The system learns a linear transformation matrix between a neutral face and a provided caricature example, then applies this operator to a new target face.
  • Relationship Exaggeration: This alters the position, size, and angle of features relative to one another using a technique called Exaggeration Difference From Mean (EDFM).

To ensure the resulting caricature maintains the subject's likeness, the system applies rigid transformations to the internal MVC of individual facial components. For lateral view synthesis, the authors approximate the head as a cylinder and introduce a novel distance-weighted modification to the MVC deformation equation, allowing the algorithm to handle self-intersecting polygons caused by perspective occlusions.

The process of transferring a 'fear' expression from an exemplar pair to a target face using MVC control polygons.

Datasets

The system's performance was evaluated using the following resources:

  • JAFFE (Japanese Female Facial Expression) Dataset: Utilized to demonstrate the system's ability to transfer and exaggerate specific facial expressions.
  • Active Appearance Models (AAM): Used to extract initial facial contours from input images.

A major advantage of this method is its "one-shot" capability; it does not require a massive training dataset of caricatures and operates effectively using a single exemplar pair (one neutral face and its corresponding hand-drawn caricature).

A sample image from the JAFFE dataset used for expression transfer experiments.

Results and Discussion

Qualitative results demonstrate that the system successfully generates caricatures that adopt the style of the provided exemplar while maintaining the distinct identity of the target subjects. Key findings include:

  • Expression Transfer and Interpolation: The MVC framework can extract an expression (such as "fear") from a source pair and map it onto a completely different neutral face, even across different genders. The intensity of the expression can be smoothly interpolated or exaggerated.
  • Lateral View Generation: The modified MVC algorithm successfully generates side-profile caricatures from flat frontal images, allowing for extreme exaggerations without breaking the image mesh.
  • Style Transfer Efficiency: By framing shape exaggeration as a linear operator, the system mimics an artist's style using only one reference pair, making it significantly more efficient than deep learning approaches requiring thousands of examples.

Conclusion

This paper presents a robust mathematical framework for caricature synthesis and expression editing. By decoupling shape and relationship exaggeration and modifying MVC to handle polygon intersections, the authors provide a tool that is both artistically flexible and identity-preserving. Future work will focus on extending this framework to video, aiming to extract and exaggerate facial expressions on moving subjects in real-time.

Summary updated

Abstract

In this paper, a novel method for caricature synthesis is developed based on mean value coordinates (MVC). Our method can be applied to any single frontal face image to learn a specified caricature face exemplar pair for frontal and side view caricature synthesis. The technique only requires one or a small number of caricature face pairs and a natural frontal face training set, while the system can transfer the style of the exemplar pair across individuals. Further exaggeration can be fulfilled in a controllable way. Our method is further extended to facial expression transfer, interpolation and exaggeration, which are
\napplications of expression editing. Moreover, the deformation equation of MVC is modified to handle the case of polygon intersections and applied to lateral view caricature synthesis from a single frontal view image. Using experiments we demonstrate that the transferred expressions are credible and the resulting caricatures can be characterized and recognized.
BibTeX
@inproceedings{yu2010caricature,
  author = {Hongchuan Yu and Jianjun Zhang},
  title = {{Caricature Synthesis Based on Mean Value Coordinates}},
  booktitle = {Bournemouth},
  year = {2010},
}
Figure