Selective Encoding for Recognizing Unreliably Localized Faces

Ang Li, Vlad I. Morariu, Larry S. DavisICCV 2015 · OpenReview

Abstract

Most existing face verification systems rely on precise face detection and registration. However, these two components are fallible under unconstrained scenarios (e.g., mobile face authentication) due to partial occlusions, pose variations, lighting conditions and limited view-angle coverage of mobile cameras. We address the unconstrained face verification problem by encoding face images directly without any explicit models of detection or registration. We propose a selective encoding framework which injects relevance information into each cluster of a descriptor codebook. An additional selector component also discards distractive image patches and improves spatial robustness. We evaluate our framework using Gaussian mixture models and Fisher vectors on challenging face verification datasets. Our framework improves robustness with no extra test time computation, demonstrating its utility in real-world mobile based active face authentication tasks.

Introduction

Standard face verification pipelines follow a strict "Detect $\rightarrow$ Align $\rightarrow$ Represent $\rightarrow$ Classify" sequence. While effective in controlled environments, this sequence often fails in real-world mobile scenarios. Factors such as extreme poses, poor lighting, or faces being partially off-screen lead to "noisy" bounding boxes that include excessive background or miss key facial features. When these unreliably localized images are processed by standard encoding schemes like Fisher Vectors (FV), background noise pollutes the facial representation, causing recognition performance to plummet. This research proposes a system inherently robust to localization errors without requiring pixel-precise detection.

Selective Encoding Framework

The proposed approach builds upon Fisher Vector encoding, which utilizes a Gaussian Mixture Model (GMM) as a "visual vocabulary" for image patches. The core innovation is the Selector component, which introduces spatial intelligence into the encoding process.

The Selective Encoding pipeline: image patches are sampled, assigned to a codebook, and then filtered by a selector based on relevance before final encoding.

The framework operates through three primary mechanisms:

  1. Codeword Relevance: During training, the system learns which GMM components typically represent facial features versus background clutter. Each component is assigned a relevance weight (0 to 1).
  2. Descriptor Selection: At test time, the system calculates the relevance of every image patch. Low-relevance patches (likely background) are discarded or down-weighted before the final Fisher Vector is calculated.
  3. Spatial Re-centering: To handle global shifts, the system "re-centers" patch coordinates based on the mean position of selected facial patches, effectively performing implicit registration mathematically even if the bounding box is off-center.

Experimental Results

The framework was evaluated on LFW (Labeled Faces in the Wild), YouTube Faces (YTF), and a large-scale Mobile Face Authentication Dataset consisting of over 19,000 video clips.

Performance comparison on LFW showing that Selective Encoding (orange) maintains higher true positive rates as localization noise (standard deviation of window offsets) increases compared to the baseline (blue).

Key findings include:

  • Robustness to Spatial Shift: On the LFW dataset with high artificial shift ($\sigma=50$ pixels), Selective Encoding maintained stable performance while the standard FV baseline saw a significant drop.
  • Video Verification: On YouTube Faces, the method consistently outperformed the original Fisher Vector across all noise levels, achieving 72.88% accuracy in high-noise scenarios compared to the baseline's 69.64%.
  • Real-World Mobile Performance: In real-world mobile tasks where detection frequently fails, Selective Encoding reduced the Equal Error Rate (EER) from 0.455 (baseline) to 0.036—a massive improvement in usability for real-world applications.

Conclusion

Selective Encoding demonstrates that visual codebooks can serve as semantic filters to mitigate the impact of background noise in unconstrained face recognition. By prioritizing facial patches and performing implicit registration, the system achieves high robustness with negligible computational overhead. While modern deep learning has shifted toward attention mechanisms, the principle of selective spatial weighting remains a foundational concept for robust computer vision in cluttered environments.

Summary updated

Abstract

Most existing face verification systems rely on precise face detection and registration. However, these two components are fallible under unconstrained scenarios (e.g., mobile face authentication) due to partial occlusions, pose variations, lighting conditions and limited view-angle coverage of mobile cameras. We address the unconstrained face verification problem by encoding face images directly without any explicit models of detection or registration. We propose a selective encoding framework which injects relevance information (e.g., foreground/background probabilities) into each cluster of a descriptor codebook. An additional selector component also discards distractive image patches and improves spatial robustness. We evaluate our framework using Gaussian mixture models and Fisher vectors on challenging face verification datasets. We apply selective encoding to Fisher vector features, which in our experiments degrade quickly with inaccurate face localization, our framework improves robustness with no extra test time computation. We also apply our approach to mobile based active face authentication task, demonstrating its utility in real scenarios.
BibTeX
@inproceedings{li2015selective,
  author = {Ang Li and Vlad I. Morariu and Larry S. Davis},
  title = {{Selective Encoding for Recognizing Unreliably Localized Faces}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=H1bz6xfObr},
}

Loading related content...

Figure