Bayesian Modeling of Facial Similarity

Baback Moghaddam, Tony Jebara, Alex PentlandNeurIPS 1998 · OpenReview

TL;DR

This paper moves face recognition from simple geometric distance (Eigenfaces) to a probabilistic framework that classifies types of image differences. By distinguishing between "intrapersonal" variations (same person, different expression) and "extrapersonal" variations (different people), the authors achieved a 95% accuracy rate, becoming the top performer in the 1996 FERET competition.

What Problem Does This Paper Address?

In the mid-90s, the "Eigenface" method was the gold standard for face recognition. It worked by projecting face images into a low-dimensional subspace and measuring the Euclidean distance between them. However, this approach has a fundamental flaw: it treats all pixel differences equally.

In reality, not all differences are created equal. A change in a person's expression or a shift in lighting can create a large "distance" in pixel space, even though the identity remains the same. Conversely, two different people might look similar in a low-dimensional projection. The authors sought to create a similarity measure that understands which variations are "incidental" (like a smile) and which are "critical" (like identity).

Methods

The core innovation is treating face recognition as a binary pattern classification problem. Instead of comparing two images directly, the authors look at the difference image ($\Delta = I_1 - I_2$).

  1. Dual Subspaces: They define two classes of variations:
  2. Intrapersonal ($\Omega_I$): Differences between images of the same individual (e.g., different expressions, lighting).
  3. Extrapersonal ($\Omega_E$): Differences between images of different individuals.
Examples of intrapersonal variations
  1. Bayesian Analysis: They use Bayes' rule to calculate the a posteriori probability $P(\Omega_I | \Delta)$—essentially asking, "Given this difference, how likely is it that these two images are the same person?"
  2. Density Estimation via PCA: Because the difference images are high-dimensional, the authors use Principal Components Analysis (PCA) to estimate the likelihood functions. They divide the vector space into a principal subspace (the top $M$ eigenvectors) and its orthogonal complement to make the math tractable.
  3. Computational Efficiency: To avoid the "costly" nonlinear computation of Bayesian similarity during a live search, the authors developed a method using whitening transformations. By pre-processing images into whitened subspace coefficients, the complex probabilistic matching reduces to a simple, fast Euclidean norm.

Datasets

The researchers evaluated their system using the ARPA FERET face database, the standard benchmark of the era.

  • Internal Testing: A subset of 74 pairs for training and 38 pairs for testing.
  • 1996 FERET Competition: A massive (for the time) double-blind test involving nearly 1,200 individuals.

Key Insights and Ideas

  • Similarity as Classification: The "aha moment" is realizing that similarity isn't a distance; it's a probability. By modeling the distribution of differences, the system learns to ignore "noise" like facial expressions.
  • The "Dual" Basis: The paper reveals that intrapersonal eigenfaces represent subtle, deformable variations (expression/lighting), while extrapersonal eigenfaces look like "standard" faces, coding for identity features like hair color, glasses, and bone structure.
Intrapersonal eigenfaces showing deformable variations
  • The ML vs. MAP Trade-off: The authors show that a simplified Maximum Likelihood (ML) approach—which only considers the intrapersonal class—is almost as accurate as the full Maximum a Posteriori (MAP) approach but twice as fast.

Key Results

  • Superior Accuracy: In the 1996 FERET competition, the Bayesian algorithm was the top performer with 95% accuracy.
  • Halving the Error: On the FERET test, the Bayesian method provided a 10% performance gain over standard Eigenfaces, effectively halving the error rate of the previous state-of-the-art.
  • Internal Benchmarks: On a smaller internal set, the Bayesian approach achieved a 90% rank-1 recognition rate, compared to 84% for standard Eigenface matching.
  • Speedup: The proposed offline transformation allows the system to perform matching using only about 400 arithmetic operations per image pair, making it viable for very large databases.

Limitations and Future Work

  • Alignment Sensitivity: While the paper mentions an automatic face-processing system for normalization (translation, scale, rotation), the Bayesian model's performance is heavily dependent on the quality of this initial geometric alignment.
  • Computational Cost of Training: While the online matching is fast, the offline training requires performing PCA on large sets of difference images, which can be computationally intensive as the training set grows.
  • Simplified ML Measure: The authors note that while the simplified ML measure is faster, it hasn't been "officially" FERET tested to the same extent as the MAP version, though internal tests show a slight 2-3% drop in accuracy.

Summary updated

Abstract

In previous work [6, 9, 10], we advanced a new technique for direct visual matching of images for the purposes of face recognition and image retrieval , using a probabilistic measure of similarity based primarily on a Bayesian (MAP) analysis of image differ(cid:173) ences, leading to a "dual" basis similar to eigenfaces [13]. The performance advantage of this probabilistic matching technique over standard Euclidean nearest-neighbor eigenface matching was recently demonstrated using results from DARPA's 1996 "FERET" face recognition competition, in which this probabilistic matching algorithm was found to be the top performer. We have further developed a simple method of replacing the costly com put ion of nonlinear (online) Bayesian similarity measures by the relatively inexpensive computation of linear (offline) subspace projections and simple (online) Euclidean norms, thus resulting in a significant computational speed-up for implementation with very large image databases as typically encountered in real-world applications.
BibTeX
@inproceedings{moghaddam1998bayesian,
  author = {Baback Moghaddam and Tony Jebara and Alex Pentland},
  title = {{Bayesian Modeling of Facial Similarity}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1998},
  url = {https://openreview.net/forum?id=Sk-dVOWdZB},
}
Figure