Image-based face recognition under illumination and pose variations

S. Kevin Zhou, Rama ChellappaJournal of the Optical Society of America A 2005 · DOI

TL;DR

This paper introduces an image-based face recognition system that tackles the challenging problem of variations in illumination and pose without relying on explicit 3D models. By deriving an illumination- and pose-invariant identity signature using subspace encoding and a Lambertian reflectance model, the approach achieves robust recognition across diverse conditions, outperforming existing methods on the PIE database.

What Problem Does This Paper Address?

Face recognition systems often struggle when images are captured under different lighting conditions or from various viewpoints. This is because these variations significantly alter the appearance of a face, making it difficult to match a probe image to a gallery image. While many existing methods focus on either illumination or pose variations independently, handling both simultaneously remains a significant challenge. The authors aim to develop a unified approach that can generalize across both illumination and pose, a crucial step towards real-world face recognition systems.

Methods

The core of the proposed method is an "image-based" approach, meaning it learns directly from 2D image data without requiring explicit 3D face models. The authors unify two existing approaches: one that handles illumination variations using a Lambertian reflectance model and subspace encoding for identity, and another that handles pose variations using a "light field" concept and subspace modeling.

The key innovation is the derivation of an illumination- and pose-invariant identity signature. This is achieved by:

  • Identity Encoding: Using subspace encoding to capture the underlying identity of a face, assuming that all human faces share similarities.
  • Illumination Modeling: Employing a Lambertian reflectance model to describe how light interacts with the face surface, allowing for generalization to unseen illumination conditions.
  • Pose Handling: Treating a given set of poses as a whole, rather than trying to generalize to arbitrary new poses.

The paper details the mathematical framework for deriving these signatures, including matrix operations for combining image data and learning basis matrices. They also address the practical issue of shadow pixels by introducing an indication vector to exclude them from the analysis.

Datasets

The primary dataset used for training and evaluation is the Pose, Illumination, and Expression (PIE) database. Specifically, the authors utilize the 'illum' subset of the PIE database, which contains images of 68 subjects under 21 illumination conditions and 13 poses. For their experiments, they select a subset of these variations to create training, gallery, and probe sets, ensuring no identity overlap between the sets.

Key Insights and Ideas

  • Unification is Key: The paper demonstrates that combining existing techniques for handling illumination and pose variations can lead to a more robust and generalizable face recognition system.
  • Illumination-Invariant Signature: The derived identity signature is shown to be invariant to illumination changes, a significant achievement for practical applications.
  • Pose as a Set: Instead of aiming for full pose generalization, the approach effectively handles a predefined set of poses, which is a pragmatic approach for many real-world scenarios.
  • Shadow Handling is Crucial: Explicitly addressing shadow pixels, which violate the Lambertian model, improves the robustness of the recognition system.
  • Illumination Easier than Pose: Experimental results suggest that handling illumination variations is generally easier than handling pose variations.

Key Results

The proposed method demonstrates strong performance on the PIE database. Key results include:

  • Scenario A (fixed gallery set):
    • Same pose, different illumination: 97% average recognition rate.
  • Different pose, same illumination: 88% average recognition rate.
  • Different pose, different illumination: 70% average recognition rate.
  • Scenario B (group of images): Recognition rates are significantly higher than in Scenario A, indicating the benefit of using multiple images.
  • Comparison with Fisher Light Field: The proposed approach achieves a "grand" average of 53% compared to 36% for the Fisher light field approach in a challenging scenario.
  • Comparison with PCA/Eigenface: The proposed method significantly outperforms baseline PCA and Eigenface approaches, with "grand" averages well below 10% for Eigenface in top-1 matches.
  • Computational Efficiency: The proposed fitting algorithm is significantly faster than the 3DMM approach, taking 1-2 seconds per image compared to 4.5 minutes.

Limitations and Future Work

  • Pose Generalization: While the approach handles a given set of poses well, generalization to arbitrary new poses is not yet implemented and is identified as future work.
  • Model Assumptions: The performance relies on the validity of the Lambertian reflectance model and the accuracy of image normalization.
  • Complex Illumination: The current model uses a Lambertian model. Future work includes exploring more complex illumination models for better approximation.
  • Nonlinear Identity Generalization: The paper assumes linear generalization for identity. Future work aims to explore nonlinear generalization.
  • Color Images: The current implementation primarily uses grayscale images, and the use of color images in the 3DMM approach is noted as a potential area for improvement.

Summary updated

Abstract

We present an image-based method for face recognition across different illuminations and poses, where the term image-based means that no explicit prior three-dimensional models are needed. As face recognition under illumination and pose variations involves three factors, namely, identity, illumination, and pose, generalizations in all these three factors are desired. We present a recognition approach that is able to generalize in the identity and illumination dimensions and handle a given set of poses. Specifically, the proposed approach derives an identity signature that is illumination- and pose-invariant, where the identity is tackled by means of subspace encoding, the illumination is characterized with a Lambertian reflectance model, and the given set of poses is treated as a whole. Experimental results using the Pose, Illumination, and Expression (PIE) database demonstrate the effectiveness of the proposed approach.
BibTeX
@article{zhou2005image,
  author = {S. Kevin Zhou and Rama Chellappa},
  title = {{Image-based face recognition under illumination and pose variations}},
  journal = {Journal of the Optical Society of America A},
  year = {2005},
  doi = {10.1364/josaa.22.000217},
}
Figure