Empirically Analyzing the Effect of Dataset Biases on Deep Face Recognition Systems

Adam Kortylewski, Bernhard Egger, Andreas Schneider, Thomas Gerig, Andreas Morel-Forster, Thomas Vetter

TL;DR

This paper introduces a method to "stress-test" deep face recognition systems by using a synthetic 3D face generator to create datasets with precisely controlled biases. The authors reveal that while modern models are surprisingly robust to lighting changes, they struggle significantly with pose variations—especially when different identities in the training set don't share the same range of motion.

What Problem Does This Paper Address?

Deep Convolutional Neural Networks (DCNNs) have achieved incredible accuracy on face recognition benchmarks. However, these benchmarks usually rely on "in the wild" datasets that lack detailed annotations for things like precise head pose, lighting angles, or facial expressions.

Because of this lack of metadata, researchers often only see a single "total recognition rate." This number is a blunt instrument: it doesn't tell you why a model failed. Did it fail because the architecture is weak, or because the training data was biased toward frontal faces? The authors argue that without fine-grained control over these "nuisance transformations," we cannot truly understand the generalization limits of our models.

Figure 1: Importance of annotated datasets for diagnosing deep face recognition systems. Left: In the wild data does not permit any analysis of the generalization ability that goes beyond the total recognition rate. Right: Our proposed synthetic face image generator enables a detailed analysis of the recognition score as a function of the most relevant nuisance transformations, such as the face pose, illumina- tion conditions, facial expressions and dataset bias.

Methods

The researchers propose a systematic empirical analysis using a fully parametric face image generator.

  1. Synthetic Generation: They use the Basel Face Model 2017 (a 3D Morphable Model) to generate synthetic identities. This allows them to perfectly control and label every variable: shape, texture, expression, 3D pose (yaw), and illumination (using spherical harmonics).
  2. Controlled Biasing: Instead of using a balanced dataset, they intentionally "break" the training data. For example, they might remove all profile views or only provide light from the left side for certain identities.
  3. Comparative Evaluation: They train standard architectures (AlexNet and VGG-16) on these biased sets and test them on a completely unbiased, "ground-truth" synthetic test set. This isolates exactly how a specific bias in training data impacts performance across different architectures.
Figure 2: Experimental setup for our empirical analysis of the effect of biased training data on the generalization ability of different DCNN architectures. (I) We generate synthetic identities with a 3D Morphable Face Model and render them in dif- ferent poses and illumination conditions. We simulate background variation by overlaying the faces on different textures. (II) We bias the training data by removing certain viewpoints from the training set. (III) We train common DCNN architectures on the biased training data. (IV) The annotation of the test data makes possible to analyze the recognition rate as a function of the face pose. It provides fine-grained information about the generalization ability of the different DCNN architectures.

Datasets

  • Synthetic Data: Generated using the Basel Face Model 2017 (BFM-2017). The authors have made their face image generator publicly available to help the community benchmark other architectures.
  • Real-World Data: The CMU Multi-PIE database was used to validate that the patterns observed in synthetic experiments (like pose sensitivity) actually hold true for real human images.

Key Insights and Ideas

  • VGG-16’s Secret Sauce: It is well-known that VGG-16 outperforms AlexNet in face recognition. This paper pinpoints why: VGG-16 is significantly better at generalizing to unseen poses. Even when trained on sparse data (e.g., only three specific angles), VGG-16 can "fill in the gaps" for intermediate poses, whereas AlexNet cannot.
  • The Lighting Paradox: DCNNs are remarkably good at generalizing to unseen illumination. A model trained only with light coming from the left can still recognize faces lit from the right with high accuracy. The authors suspect this is because the models focus on image gradients, which remain somewhat consistent even as light shifts.
  • The Disentanglement Failure: A major finding is that current DCNNs struggle to "disentangle" identity from pose. If Identity A is only shown looking left and Identity B is only shown looking right during training, the model fails to realize that "looking left" is a pose, not a feature of Identity A. It essentially bakes the pose into the identity representation.

Key Results

  • Pose Sensitivity: When training was restricted to a yaw range of [-45°, 45°], recognition rates plummeted for faces outside that range. VGG-16 maintained a Total Recognition Rate (TRR) of ~86% compared to AlexNet’s ~78% in these scenarios.
  • Sparse Sampling: In an experiment where models were trained on only three poses (-45°, 0°, 45°), VGG-16 achieved a TRR of 81.9%, while AlexNet lagged at 69.3%. VGG-16 showed a nearly perfect recognition rate across the entire range, demonstrating superior architectural generalization.
  • Illumination Robustness: Both models achieved >93% accuracy on unseen light directions, even when the training light was restricted to a single hemisphere.
Effect of restricting the range of yaw poses at training time.

Limitations and Future Work

The authors acknowledge that their study used a relatively small number of identities (30) compared to the millions found in modern production systems. Additionally, their synthetic lighting model is simplified and does not account for complex effects like self-shadowing or hard shadows, which might overstate the models' robustness to real-world lighting.

Future work suggested includes investigating the specific neural mechanisms that allow VGG-16 to generalize better and exploring new architectural constraints that could help models better disentangle pose from identity across different individuals.

Summary updated

Abstract

It is unknown what kind of biases modern in the wild face datasets have because of their lack of annotation. A direct consequence of this is that total recognition rates alone only provide limited insight about the generalization ability of a Deep Convolutional Neural Networks (DCNNs). We propose to empirically study the effect of different types of dataset biases on the generalization ability of DCNNs. Using synthetically generated face images, we study the face recognition rate as a function of interpretable parameters such as face pose and light. The proposed method allows valuable details about the generalization performance of different DCNN architectures to be observed and compared. In our experiments, we find that: 1) Indeed, dataset bias has a significant influence on the generalization performance of DCNNs. 2) DCNNs can generalize surprisingly well to unseen illumination conditions and large sampling gaps in the pose variation. 3) Using the presented methodology we reveal that the VGG-16 architecture outperforms the AlexNet architecture at face recognition tasks because it can much better generalize to unseen face poses, although it has significantly more parameters. 4) We uncover a main limitation of current DCNN architectures, which is the difficulty to generalize when different identities to not share the same pose variation. 5) We demonstrate that our findings on synthetic data also apply when learning from real-world data. Our face image generator is publicly available to enable the community to benchmark other DCNN architectures.
BibTeX
@inproceedings{kortylewski2017empirically,
  author = {Adam Kortylewski and Bernhard Egger and Andreas Schneider and Thomas Gerig and Andreas Morel-Forster and Thomas Vetter},
  title = {{Empirically Analyzing the Effect of Dataset Biases on Deep Face Recognition Systems}},
  booktitle = {Workshops @ CVPR},
  year = {2017},
  doi = {10.1109/cvprw.2018.00283},
}
Figure