Establishing Good Benchmarks and Baselines for Face Recognition

Nicolas Pinto, James J. DiCarlo, David CoxINRIA 2008

TL;DR

This paper argues that many existing face recognition benchmarks might be too easy, allowing simple algorithms to achieve high performance by exploiting dataset regularities rather than solving the core problem of face recognition. The authors propose a rudimentary "V1-like" baseline system and evaluate it on several standard datasets and the new Labeled Faces in the Wild (LFW) dataset, suggesting LFW offers a more challenging and realistic benchmark.

What Problem Does This Paper Address?

Face recognition in real-world images is a notoriously difficult problem due to significant variations in pose, lighting, expression, occlusion, and background. To accurately assess progress in this field, researchers rely on benchmark datasets to evaluate algorithms. However, the authors raise a critical concern: are these benchmarks truly representative of the real-world challenges, or do they contain "cheats" – trivial regularities that allow even simple systems to perform well? They draw a parallel to object recognition, where a basic system inspired by early visual processing achieved surprisingly high scores on a popular dataset, while failing on a simpler synthetic task designed to better capture real-world variations. This suggests that high performance on some datasets might not reflect true progress in robust face recognition. The paper aims to establish a more meaningful baseline for face recognition by evaluating a simple system on existing benchmarks and the challenging LFW dataset.

Methods

The core of the paper's methodology is the creation and evaluation of simple baseline models. They propose three representations:

  • Raw Grayscale Pixels: A straightforward representation where images are converted to grayscale, resized, and flattened into a vector.
  • "V1-like" Representation: This model is inspired by the known properties of the primary visual cortex (V1). It involves locally normalizing image patches and applying a set of Gabor filters tuned to various orientations and spatial frequencies. This is a "null model" in the sense that it doesn't incorporate sophisticated shape representations or explicit mechanisms for handling image variation.
  • "V1-like+" Representation: This extends the "V1-like" model by adding a collection of easily computed additional features, such as histograms.

These representations are then used with a linear Support Vector Machine (SVM) classifier. For most standard datasets, dimensionality reduction via Principal Component Analysis (PCA) is applied before classification. For the LFW dataset, a slightly different approach is used, involving computing element-wise squared differences between image representations for face pairs.

Datasets

The paper evaluates its baseline models on the following datasets:

  • Olivetti Research Lab (ORL) dataset: 40 subjects, 10 grayscale images per subject.
  • Yale dataset: 15 subjects, 11 grayscale images per subject.
  • AR (Aleix and Robert) dataset: Over 4,000 color images of 126 subjects, with variations in expression, lighting, and occlusion.
  • Computer Vision Laboratory (CVL) dataset: 114 subjects, 7 color images per subject, with variations in expression and pose.
  • Labeled Faces in the Wild (LFW) dataset: 13,233 images of 5,749 individuals, described as "unconstrained" with significant natural variation. This is a newly introduced benchmark for the authors' study.

Additionally, a synthetic face dataset was created using 3D face meshes to parametrically control variations in view, lighting, and scale.

Key Insights and Ideas

  • The "Easy" Benchmarks Problem: The paper strongly suggests that many established face recognition datasets (ORL, Yale, AR, CVL) are too easily solved by simple, low-level feature extractors like the "V1-like" model. This raises questions about whether performance reported on these datasets truly reflects progress in robust face recognition.
  • "V1-like" as a Powerful Baseline: The "V1-like" model, despite its simplicity and lack of explicit variation tolerance mechanisms, achieves remarkably high performance (often near-perfect) on the older datasets. This highlights the potential for simple cues to dominate performance on these benchmarks.
  • LFW as a More Realistic Challenge: The LFW dataset, with its "unconstrained" nature, proves to be a more significant challenge. Even the "V1-like" model, which excels on older datasets, achieves only around 68% accuracy on LFW, indicating that it doesn't fully exploit trivial regularities and leaves substantial room for improvement.
  • Synthetic Data for Controlled Evaluation: The synthetic dataset demonstrates that when variations are precisely controlled and increased, the simple baseline models rapidly decline in performance, confirming their inability to generalize beyond specific dataset biases.

Key Results

  • On the ORL dataset, the "V1-like" model achieved perfect performance with 8 training examples per subject.
  • On the Yale dataset, the "V1-like" model achieved near-perfect performance (>99%) with 8 training examples.
  • On the AR dataset, the "V1-like" model achieved greater than 98% performance with 8 training examples.
  • On the CVL dataset, the "V1-like" model achieved better than 90% correct on a protocol using only frontal views, but performance dropped to ~50% correct on a protocol using all views.
  • On the LFW dataset, the "V1-like" model achieved ~68% correct performance, which is substantially higher than a pure chance baseline (50%) but significantly lower than 100%.
  • On the synthetic dataset, performance of all baseline models rapidly declined as increasing amounts of view variation were introduced.

Limitations and Future Work

  • Dataset Bias: The primary limitation identified is the potential for existing face recognition datasets to contain low-level regularities that are exploited by simple algorithms, leading to inflated performance metrics.
  • "V1-like" Model Limitations: While effective as a baseline, the "V1-like" model is not presented as a state-of-the-art face recognition system. Its performance on LFW suggests it captures some aspects of the problem but not all.
  • Future Work: The authors suggest that performance reports on LFW should be considered with the ~70% baseline in mind, implying that models should aim for substantially higher performance to demonstrate genuine progress. They also implicitly advocate for the development of more challenging and representative benchmarks.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{pinto2008establishing,
  author = {Nicolas Pinto and James J. DiCarlo and David Cox},
  title = {{Establishing Good Benchmarks and Baselines for Face Recognition}},
  booktitle = {INRIA},
  year = {2008},
}
Figure