Towards Improved Cartoon Face Detection and Recognition Systems

Saurav Jha, Nikhil Agarwal, Suneeta AgarwalarXiv 2018 · OpenReview

Abstract

Given the recent deep learning advancements in face detection and recognition techniques for human faces, this paper answers the question "how well would they work for cartoons?" - a domain that remains largely unexplored until recently, mainly due to the unavailability of large scale datasets and the failure of traditional methods on these. Our work studies and extends multiple frameworks for the aforementioned tasks. For face detection, we incorporate the Multi-task Cascaded Convolutional Network (MTCNN) architecture and contrast it with conventional methods. For face recognition, our two-fold contributions include: (i) an inductive transfer learning approach combining the feature learning capability of the Inception v3 network and the feature recognizing capability of Support Vector Machines (SVMs), (ii) a proposed Hybrid Convolutional Neural Network (HCNN) framework trained over a fusion of pixel values and 15 manually located facial keypoints. All the methods are evaluated on the Cartoon Faces in the Wild (IIIT-CFW) database. We demonstrate that the HCNN model offers stability superior to that of Inception+SVM over larger input variations, and explore the plausible architectural principles. We show that the Inception+SVM model establishes a state-of-the-art F1 score on the task of gender recognition of cartoon faces. Further, we introduce a small database hosting location coordinates of 15 points on the cartoon faces belonging to 50 public figures of the IIIT-CFW database.

Introduction

While human face recognition has reached high levels of maturity, cartoon faces present a unique set of challenges. Cartoons often feature exaggerated attributes—unnatural skin tones, anomalous facial outlines, and a blatant disregard for symmetry—that cause traditional computer vision models to fail. The primary hurdles have been a lack of large-scale, annotated datasets and the fact that "hand-crafted" features (like Haar cascades) are not robust enough to handle the vast artistic variety found in comics and animation. This research bridges this gap by leveraging deep learning to create more stable detection and recognition systems specifically for the cartoon domain.

Methodology

Face Detection

The study implements the Multi-task Cascaded Convolutional Network (MTCNN), a deep learning framework that performs face detection and alignment simultaneously. This approach is compared against traditional methods such as Histogram of Oriented Gradients (HOG) and Haar-feature-based detectors.

Face Recognition

Two distinct neural network strategies are proposed for character and gender recognition:

1. Inductive Transfer Learning: A pre-trained Inception v3 model (originally trained on ImageNet) is utilized as a feature extractor. Instead of the standard softmax output, the 2048-dimensional "bottleneck" features are fed into Support Vector Machines (SVM) and Gradient Boosting (GB) classifiers.

Inception v3 based transfer learning architecture for cartoon face recognition.

2. Hybrid Convolutional Neural Network (HCNN): This custom architecture processes two inputs in parallel: raw pixel values and the coordinates of 15 facial keypoints (landmarks like eye corners, nose tip, and lip centers). The model utilizes an auxiliary classifier during training to improve stability and convergence.

The proposed recognition framework (Phase I & II) and the detailed HCNN classifier architecture.

Experimental Setup and Datasets

The primary benchmark used is the IIIT-CFW (Cartoon Faces in the Wild) database, containing 8,928 images of 100 public figures. For pre-training and landmark detection, the CASIA WebFace and Kaggle Facial Keypoints datasets were employed. A significant contribution of this work is the manual annotation of 15 facial keypoints for 750 images across 50 characters from the IIIT-CFW set to facilitate the hybrid approach.

Results and Discussion

  • Detection Performance: MTCNN outperformed traditional methods, achieving a True Positive Rate (TPR) of 78.17% on randomly chosen cartoon images, surpassing the previous state-of-the-art of 74.2%.
  • Gender Recognition: The Inception v3 + SVM model established a new state-of-the-art F1-measure of 0.910.
  • Character Recognition: On a 100-class problem, the HCNN (using pixel data + keypoints) achieved a top-5 error rate of 25.10%, outperforming models relying on pixels alone.
  • The Role of Landmarks: Incorporating facial keypoints as a secondary input to the CNN resulted in a 5.87% gain in the top-5 error rate, suggesting that relative geometry remains a vital signal even in stylized art.

Conclusion

This work demonstrates that deep learning architectures can be successfully adapted to the cartoon domain. While the Inception v3 + SVM approach excels in simpler tasks like gender recognition, the custom HCNN shows better stability as character classes increase. Future work will focus on expanding artistic style diversity and improving landmark detection resilience to the "anomalous organ shapes" frequently found in caricatures.

Summary updated

Abstract

Given the recent deep learning advancements in face detection and recognition techniques for human faces, this paper answers the question "how well would they work for cartoons'?" - a domain that remains largely unexplored until recently, mainly due to the unavailability of large scale datasets and the failure of traditional methods on these. Our work studies and extends multiple frameworks for the aforementioned tasks. For face detection, we incorporate the Multi-task Cascaded Convolutional Network (MTCNN) architecture and contrast it with conventional methods. For face recognition, our two-fold contributions include: (i) an inductive transfer learning approach combining the feature learning capability of the Inception v3 network and the feature recognizing capability of Support Vector Machines (SVMs), (ii) a proposed Hybrid Convolutional Neural Network (HCNN) framework trained over a fusion of pixel values and 15 manually located facial keypoints. All the methods are evaluated on the Cartoon Faces in the Wild (IIIT-CFW) database. We demonstrate that the HCNN model offers stability superior to that of Inception+SVM over larger input variations, and explore the plausible architectural principles. We show that the Inception+SVM model establishes a state-of-the-art F1 score on the task of gender recognition of cartoon faces. Further, we introduce a small database hosting location coordinates of 15 points on the cartoon faces belonging to 50 public figures of the IIIT-CFW database.
BibTeX
@misc{jha2018towards,
  author = {Saurav Jha and Nikhil Agarwal and Suneeta Agarwal},
  title = {{Towards Improved Cartoon Face Detection and Recognition Systems}},
  year = {2018},
  eprint = {1804.01753},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.01753},
}
Figure