Variational Prototype Learning for Deep Face Recognition

Jiankang Deng, Jia Guo, Jing Yang, Alexandros Lattas, Stefanos Zafeiriou

Introduction

Modern face recognition systems have achieved remarkable success by mapping facial images into a high-dimensional latent space where identities are separated by margin-based softmax losses. However, these methods typically represent each class as a single static point, or "prototype." This paper introduces Variational Prototype Learning (VPL), a novel framework that moves beyond point representations to model each facial identity as a distribution. By injecting "memorized" features from previous training steps into class prototypes, VPL effectively simulates sample-to-sample comparisons, significantly enhancing the discriminative power of deep recognition models.

The Limitation of Standard Prototype Learning

Traditional Prototype Learning (PL) focuses on sample-to-prototype comparisons. During training, the model forces image embeddings to align with their respective class prototypes while pushing them away from others. While effective, this approach ignores the complex intra-class variations—such as changes in pose, lighting, and age—that exist between individual samples.

Comparison of sample-to-prototype vs sample-to-sample similarities and their distributions.

As shown in the figure above, sample-to-prototype similarities often exhibit a much narrower and higher distribution than sample-to-sample similarities. This can create an "illusion" of a perfect embedding where the loss is low, but the model fails to generalize to the rich diversity of real-world facial data. This issue is particularly acute for "tail" classes with very few images, where the prototype can easily overfit to a limited set of samples.

Methodology: Variational Prototype Learning

VPL addresses these shortcomings by treating each class as a variational distribution rather than a fixed vector. The core of the method lies in three key components:

  1. Feature Memory Bank: The system maintains a memory bank that stores recent embedding features for every class.
  2. Slow Feature Drift: The authors observed that after an initial training phase, the features produced by the network change very slowly. This allows "old" features from previous iterations to serve as valid samples of the current class distribution without requiring complex generative modeling.
  3. Variational Prototypes: During the forward pass, a "variational prototype" is constructed by combining the standard learned prototype with a randomly selected feature from that class’s memory bank.
Conceptual comparison between standard Prototype Learning and the proposed Variational Prototype Learning.

By comparing a current sample against these variational prototypes, the model is forced to account for intra-class variance. This "plug-and-play" module can be easily integrated into existing loss functions like ArcFace, resulting in VPL-ArcFace. Notably, VPL adds negligible computational overhead, requiring only minimal extra GPU memory.

Key Results and Insights

VPL consistently outperforms state-of-the-art methods across a wide range of benchmarks, demonstrating its robustness and efficiency:

  • Demographic Robustness: On the InsightFace Recognition Test (IFRT), VPL-ArcFace improved accuracy on the Asian demographic group by an absolute 3.83% (from 37.27% to 41.10%), showcasing its ability to handle difficult or underrepresented data.
  • Long-Tail Performance: VPL outperformed specialized methods like AdaptiveFace by 1.32% on datasets where identities have fewer than 20 images.
  • Large-Scale Benchmarks: The method set new high bars on IJB-C (96.76% TAR@FAR=1e-4) and MegaFace (98.80% rank-1 identification), surpassing strong competitors like BroadFace and CurricularFace.

Conclusion

Variational Prototype Learning provides a simple yet powerful way to incorporate intra-class variation into the training of face recognition models. By leveraging the slow drift of features to sample class distributions, VPL encourages the model to explore the feature space more thoroughly and learn more robust embeddings. This approach is particularly effective for large-scale, open-set classification tasks where "tail" distributions and high intra-class variance are common.

Summary updated

Abstract

Deep face recognition has achieved remarkable improvements due to the introduction of margin-based softmax loss, in which the prototype stored in the last linear layer represents the center of each class. In these methods, training samples are enforced to be close to positive prototypes and far apart from negative prototypes by a clear margin. However, we argue that prototype learning only employs sample-to-prototype comparisons without considering sample-to-sample comparisons during training and the low loss value gives us an illusion of perfect feature embedding, impeding the further exploration of SGD. To this end, we propose Variational Prototype Learning (VPL), which represents every class as a distribution instead of a point in the latent space. By identifying the slow feature drift phenomenon, we directly inject memorized features into prototypes to approximate variational prototype sampling. The proposed VPL can simulate sample-to-sample comparisons within the classification framework, encouraging the SGD solver to be more exploratory, while boosting performance. Moreover, VPL is conceptually simple, easy to implement, computationally efficient and memory saving. We present extensive experimental results on popular benchmarks, which demonstrate the superiority of the proposed VPL method over the state-of-the-art competitors.
BibTeX
@inproceedings{deng2021variational,
  author = {Jiankang Deng and Jia Guo and Jing Yang and Alexandros Lattas and Stefanos Zafeiriou},
  title = {{Variational Prototype Learning for Deep Face Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.01173},
}
Figure