Deep Learning Based Single Sample Per Person Face Recognition: A Survey

Delong Chen, Fan Liu, Zewen LiarXiv 2020 · DOI · OpenReview

TL;DR

This paper surveys deep learning approaches for single-sample face recognition (SSPR), a challenging problem where only one image per person is available for training. It categorizes existing methods into "virtual sample" and "generic learning" techniques, providing a comprehensive overview of their strengths, weaknesses, and future directions, including unsupervised learning and addressing the semantic gap.

What Problem Does This Paper Address?

Traditional face recognition systems often struggle when faced with the "Single Sample Per Person" (SSPP) or "one-shot" scenario. This occurs in practical applications like identity card verification or access control where only a single reference image exists for each individual. Deep learning models, while powerful, are data-hungry and tend to overfit when trained on such limited data. Existing surveys often focus on traditional methods, leaving a gap in understanding the rapidly evolving deep learning landscape for SSPR. This paper aims to bridge that gap by systematically reviewing and categorizing deep learning-based SSPR methods.

Methods

The paper categorizes deep learning-based SSPR methods into two main families:

  • Virtual Sample Methods: These approaches aim to artificially increase the size and diversity of the training data. They are further divided into:
  • Virtual Image Generation: Creating new, synthetic face images that resemble the original identity but with variations (e.g., pose, expression). Techniques include Autoencoders and Generative Adversarial Networks (GANs).
  • Virtual Feature Generation: Generating synthetic feature representations rather than full images, aiming to preserve identity information while introducing variations.
  • Generic Learning Methods: These methods leverage an additional, larger, multi-sample dataset (a "generic set") to aid in training. The paper breaks these down into three sub-categories:
  • Combining Traditional Methods and Deep Features: Integrating established techniques like sparse representation or SVMs with features extracted by deep networks.
  • Improving Loss Functions: Modifying or designing new loss functions (e.g., softmax loss, triplet loss) to better handle the imbalanced nature of SSPR.
  • Improving Network Structure: Designing novel network architectures or incorporating auxiliary tasks (like reconstruction) to enhance feature robustness and disentangle identity from variations.

Datasets

The paper reviews several commonly used datasets for evaluating SSPR methods:

  • AR Database: Contains over 2600 grayscale images of 100 subjects with variations in expression, illumination, and occlusions.
  • Extended Yale B Database: Features 5760 grayscale images of 10 subjects with variations in illumination and pose.
  • Labeled Faces in the Wild (LFW) Dataset: A large-scale, unconstrained dataset with 13,233 RGB images of 5749 subjects, known for its complex facial variations.
  • MS-Celeb-1M Dataset: A massive dataset released by Microsoft, containing web-collected RGB images of one million subjects, often used for multi-sample face recognition but also relevant for SSPR evaluation.

The paper notes that for SSPR, the "gallery set" typically contains only one sample per person, and often a neutral frontal face is chosen as the gallery image.

Key Insights and Ideas

  • Virtual Samples are a Double-Edged Sword: While virtual sample generation (especially with GANs) can introduce valuable intra-class variation, it's crucial to ensure identity information is preserved. Incorrect generation can harm model performance.
  • Generic Learning Needs Domain Adaptation: When using external datasets, a significant "domain gap" can exist between the generic set and the target single-sample set. Addressing this domain adaptation is critical for effective transfer learning.
  • Loss Functions Matter: The choice and design of loss functions play a vital role in adapting deep models to the scarcity of data in SSPR, particularly in handling class imbalance and preventing overfitting.
  • Network Architecture Can Help: Beyond loss functions, modifying network structures with auxiliary tasks like reconstruction can improve feature robustness and disentangle identity from variations.
  • The Semantic Gap is a Fundamental Challenge: Current deep learning models often lack a deep understanding of semantic information (like gender, age, ethnicity) that humans naturally use for recognition. Bridging this "semantic gap" is a promising avenue for future research.

Key Results

The paper presents performance evaluations on the AR, Extend Yale B, LFW, and MS-Celeb-1M datasets. While specific quantitative results are detailed in Table 1, some highlights include:

  • High Accuracy Achieved: Deep learning-based methods have achieved impressive accuracies, with some reaching over 99% on datasets like AR and MS-Celeb-1M.
  • Combining Traditional and Deep Methods is Effective: Methods that combine traditional techniques with deep features often show strong performance, particularly on datasets with complex variations like AR and LFW. For instance, Liu et al. (2019) achieved 98.50% on AR, and Wang et al. (2019) achieved 99.29% on LFW.
  • 3DMM-CNNs Offer Speed Advantages: For virtual sample generation, 3DMM-CNN models were found to require significantly less processing time compared to other methods while achieving similar performance.

Limitations and Future Work

The authors identify several key areas for future research:

  • Domain Adaptation: This remains a critical challenge, especially for heterogeneous face recognition scenarios. More research is needed to adapt existing deep domain adaptation techniques to the SSPR context.
  • Large-Scale Unsupervised Pre-training: Given the need for robust face representations, leveraging large amounts of unlabeled data through unsupervised pre-training is highlighted as a promising direction. This could bypass the need for identity labels in additional datasets.
  • Addressing the Semantic Gap: The paper emphasizes that current models lack the semantic understanding that humans possess. Future work should explore incorporating semantic information and learning from human visual perception mechanisms.
  • Theoretical Guarantees for Virtual Samples: While virtual sample methods have shown empirical success, there's a lack of theoretical understanding regarding their effectiveness and optimal usage (e.g., how many samples are needed).
  • Novel Metrics: Developing objective metrics to evaluate the quality and effectiveness of generated virtual samples is crucial for fair comparison and advancement in the field.

Summary updated

Abstract

Face recognition has long been an active research area in the field of artificial intelligence, particularly since the rise of deep learning in recent years. In some practical situations, each identity has only a single sample available for training. Face recognition under this situation is referred to as single sample face recognition and poses significant challenges to the effective training of deep models. Therefore, in recent years, researchers have attempted to unleash more potential of deep learning and improve the model recognition performance in the single sample situation. While several comprehensive surveys have been conducted on traditional single sample face recognition approaches, emerging deep learning based methods are rarely involved in these reviews. Accordingly, we focus on the deep learning-based methods in this paper, classifying them into virtual sample methods and generic learning methods. In the former category, virtual images or virtual features are generated to benefit the training of the deep model. In the latter one, additional multi-sample generic sets are used. There are three types of generic learning methods: combining traditional methods and deep features, improving the loss function, and improving network structure, all of which are covered in our analysis. Moreover, we review face datasets that have been commonly used for evaluating single sample face recognition models and go on to compare the results of different types of models. Additionally, we discuss problems with existing single sample face recognition methods, including identity information preservation in virtual sample methods, domain adaption in generic learning methods. Furthermore, we regard developing unsupervised methods is a promising future direction, and point out that the semantic gap as an important issue that needs to be further considered.
BibTeX
@misc{chen2020deep,
  author = {Delong Chen and Fan Liu and Zewen Li},
  title = {{Deep Learning Based Single Sample Per Person Face Recognition: A Survey}},
  year = {2020},
  eprint = {2006.11395},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.11395},
}
Figure