Generalizing Face Representation with Unlabeled Data

Yichun Shi, Anil K. Jain

Abstract

In recent years, significant progress has been made in face recognition, which can be partially attributed to the availability of large-scale labeled face datasets. However, since the faces in these datasets usually contain limited degree and types of variation, the resulting trained models generalize poorly to more realistic unconstrained face datasets. While collecting labeled faces with larger variations could be helpful, it is practically infeasible due to privacy and labor cost. In comparison, it is easier to acquire a large number of unlabeled faces from different domains, which could be used to regularize the learning of face representations. We present an approach to use such unlabeled faces to learn generalizable face representations, where we assume neither the access to identity labels nor domain labels for unlabeled images. Experimental results on unconstrained datasets show that a small amount of unlabeled data with sufficient diversity can (i) lead to an appreciable gain in recognition performance and (ii) outperform the supervised baseline when combined with less than half of the labeled data. Compared with the state-of-the-art face recognition methods, our method further improves their performance on challenging benchmarks, such as IJB-B, IJB-C and IJB-S.

Introduction: The Domain Gap in Face Recognition

Most state-of-the-art face recognition (FR) systems are trained on massive datasets of web-crawled celebrity photos, such as MS-Celeb-1M. While these datasets are extensive, they are often "semi-constrained"—characterized by decent lighting, mostly frontal poses, and high image quality. When these models are deployed in real-world scenarios—such as grainy surveillance footage, extreme head angles, or heavy motion blur—their performance drops significantly. This phenomenon is known as the domain gap.

Overview of the proposed framework leveraging unlabeled data to generalize face recognition to unconstrained environments.

Collecting and manually labeling millions of "unconstrained" real-world photos is prohibitively expensive and raises significant privacy concerns. To address this, researchers from Michigan State University have developed a semi-supervised framework that leverages the abundance of unlabeled diverse images to teach models to recognize faces regardless of the environment.

Proposed Methodology

The proposed approach utilizes a two-pronged semi-supervised strategy to bridge the domain gap by learning domain-agnostic representations.

1. Adversarial Domain Alignment

The system employs a Discriminator ($D$) that attempts to distinguish whether a face feature originated from the labeled "clean" dataset or the unlabeled "messy" dataset. The main Embedding Network ($f$) is trained to "fool" this discriminator. This adversarial process forces the embedding network to extract features that are consistent across both domains, effectively aligning the feature distributions.

2. Multi-mode Augmentation Network ($G$)

The core innovation is a style-transfer-inspired Augmentation Network ($G$). This network learns the "visual styles" (e.g., blur, lighting, noise) present in the unlabeled data. It can then take a labeled celebrity photo and "augment" it with these difficult styles while strictly preserving the person's identity. This allows the model to practice recognizing the same identity across a vast variety of simulated environments.

The semi-supervised training pipeline featuring the Augmentation Network (G), Embedding Network (f), and Discriminator (D).

Unlike standard data augmentation (which uses pre-defined filters like "rotate" or "crop"), this method is data-driven—it discovers the difficult conditions directly from the unlabeled data. By using a reconstruction loss and a style encoder, the network ensures that while the texture changes, the geometry and identity of the face remain intact.

Experimental Results

The framework was evaluated using MS-Celeb-1M for labeled training and WiderFace (containing ~70K diverse, unconstrained faces) for unlabeled training.

Performance on Unconstrained Benchmarks

  • IJB-C: On this challenging "in-the-wild" benchmark, the proposed method achieved a True Accept Rate (TAR) of 77.39% at a False Accept Rate (FAR) of 1e-7, a substantial improvement over the supervised baseline of 62.90%.
  • IJB-S (Surveillance Mastery): On the notoriously difficult surveillance-focused dataset, the method improved Rank-1 accuracy from 53.23% (baseline) to 59.29%, setting a new state-of-the-art at the time.

Efficiency and Diversity

A key finding of the study is that diversity trumps quantity. The researchers demonstrated that a model trained on 1 million labeled images plus their unlabeled augmentation outperformed a baseline trained on 4 million labeled images. This suggests that a small amount of highly diverse unlabeled data is more effective for generalization than millions of additional standard labeled images.

Conclusion

This research demonstrates that unlabeled data can be a powerful tool for regularizing face representations. By combining adversarial alignment with data-driven augmentation, the model learns to be domain-agnostic rather than domain-specific. While the augmentation network can sometimes produce visual artifacts, the resulting features are significantly more robust to the variations found in real-world, unconstrained environments.

Summary updated

Abstract

In recent years, significant progress has been made in face recognition, which can be partially attributed to the availability of large-scale labeled face datasets. However, since the faces in these datasets usually contain limited degree and types of variation, the resulting trained models generalize poorly to more realistic unconstrained face datasets. While collecting labeled faces with larger variations could be helpful, it is practically infeasible due to privacy and labor cost. In comparison, it is easier to acquire a large number of unlabeled faces from different domains, which could be used to regularize the learning of face representations. We present an approach to use such unlabeled faces to learn generalizable face representations, where we assume neither the access to identity labels nor domain labels for unlabeled images. Experimental results on unconstrained datasets show that a small amount of unlabeled data with sufficient diversity can (i) lead to an appreciable gain in recognition performance and (ii) outperform the supervised baseline when combined with less than half of the labeled data. Compared with the state-of-the-art face recognition methods, our method further improves their performance on challenging benchmarks, such as IJB-B, IJB-C and IJB-S.
BibTeX
@inproceedings{shi2021generalizing,
  author = {Yichun Shi and Anil K. Jain},
  title = {{Generalizing Face Representation with Unlabeled Data}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=aJdhnw-5jU},
}
Figure