A Bayesian Scene-Prior-Based Deep Network Model for Face Verification

Huafeng Wang, Wenfeng Song, Wanquan Liu, Ning Song, Yuehai Wang, Haixia Pan2018 · DOI

TL;DR

This paper introduces a novel Bayesian scene-prior-based deep network model for face verification that tackles the challenge of limited training data. By leveraging scene semantics and a Bayesian approach to model scene variations, the method effectively augments training data and improves verification accuracy, achieving state-of-the-art results on challenging benchmarks.

What Problem Does This Paper Address?

Face verification, a critical task in computer vision, has seen significant performance improvements with deep learning. However, these methods typically require vast amounts of labeled training data. A major challenge arises when dealing with scenarios where only a few samples per person are available, leading to a significant drop in performance. This is because deep learning models struggle to capture the full variability of a person's appearance across different scenes and conditions with limited data. The authors aim to address this "small number of training samples" problem by incorporating scene information as a prior to improve face verification, even with scarce training data.

Methods

The core of the proposed method is a Bayesian scene-prior-based deep network model. The key ideas are:

  • Scene as a Prior: Instead of solely focusing on facial features, the model treats the "scene" or background as a crucial prior for face verification. The intuition is that a person's appearance can vary significantly due to different backgrounds, and by modeling these scene variations, we can better understand the intrinsic features of an individual.
  • Scene Inference and Generation: The model uses a Bayesian approach to learn scene distributions. It infers possible scenes for a given face and then uses this information to generate new, augmented face images. This process effectively expands the training dataset by creating variations that mimic different scene conditions.
  • CRF-RNN Integration: Conditional Random Fields (CRFs) are integrated with Recurrent Neural Networks (RNNs) to extract semantic scene information from face images. This allows for a more robust understanding of the background context.
  • Deep Network Training: The augmented dataset, enriched with scene variations, is then used to train a deep convolutional neural network (CNN) for face verification. The model leverages both global and local feature distributions, aiming to mimic human cognitive processes that first understand scenes and then identify objects within them.
  • Hyperparameter Optimization: A mixture Gaussian model is used to handle noisy scenes, and a combined energy function (network error + scene error) is optimized to improve robustness.

Datasets

  • Training: CASIA-WebFace dataset (5749 identities, 10,575 subjects, 494,414 face images) was used for training the model and learning the scene dictionary.
  • Evaluation:
    • LFW (Labeled Faces in the Wild) dataset: Used for validation, specifically the "view #2 subset" which contains 6000 pairs.
  • YTF (YouTube Faces) database: Used for evaluation, containing 3425 videos of 1595 different subjects.

Key Insights and Ideas

  • Scene is as Important as Face: The paper strongly suggests that the background scene plays a significant role in face verification, especially when training data is limited. By modeling scene variations, we can achieve better generalization.
  • Data Augmentation via Scene Transfer: The core innovation is using scene semantics to generate new training samples, effectively performing a form of domain transfer learning to overcome data scarcity.
  • Bayesian Priors for Robustness: The Bayesian framework allows for modeling uncertainty in scene distributions, leading to a more robust model that is less susceptible to variations in lighting, pose, and expression.
  • Human-like Recognition: The approach aims to mimic human visual perception, where understanding the context (scene) aids in recognizing objects (faces).

Key Results

  • LFW Dataset: Achieved an accuracy of 99.2% on the LFW dataset (view #2 subset) under an unsupervised protocol, outperforming previous literature results (99.12% with an unsupervised protocol).
  • YTF Dataset: Achieved an accuracy of 94.3% on the YouTube Faces database, also outperforming literature results (93.2% with an unsupervised protocol).
  • Generalization: The model demonstrates strong generalization capabilities, performing well on unseen data and outperforming other state-of-the-art models, particularly in scenarios with limited training data.
  • Efficiency: The proposed model requires significantly less training data compared to methods like DeepFace, achieving comparable or better performance. For instance, it used only 1/8th the training data of DeepFace.
  • ROC Curves: The ROC curves show competitive performance against other leading models on both LFW and YTF datasets, with the proposed model achieving the best AUC on YTF and a very competitive AUC on LFW.

Limitations and Future Work

  • Failure Cases: The paper acknowledges that the model still fails on some face pairs, likely due to significant variations in facial expressions, shelter, and other factors.
  • Future Work: The authors plan to conduct further research on facial expression scenes and aim to make their model capable of transforming all facial expression scenes into uniform scenes.

Summary updated

BibTeX
@misc{wang2018bayesian,
  author = {Huafeng Wang and Wenfeng Song and Wanquan Liu and Ning Song and Yuehai Wang and Haixia Pan},
  title = {{A Bayesian Scene-Prior-Based Deep Network Model for Face Verification}},
  year = {2018},
  doi = {10.3390/s18061906},
}
Figure