Teacher Guided Neural Architecture Search for Face Recognition

Xiaobo WangAAAI 2021 · DOI

Introduction

Knowledge Distillation (KD) has become a cornerstone for deploying deep learning models on resource-constrained devices. By training a compact "student" network to mimic a high-performance "teacher," researchers can achieve significant reductions in model size and latency. However, a critical bottleneck remains: the manual design of the student architecture. Traditionally, researchers rely on hand-crafted heuristics, such as selecting a standard ResNet-18, which may not be optimal for the specific demands of face recognition—an "open-set" problem requiring robust feature embeddings for unseen identities.

This paper introduces Teacher Guided Neural Architecture Search (TNAS), an automated framework that leverages the teacher model's expertise to search for the optimal student architecture. By optimizing network width and depth simultaneously, TNAS discovers compact models that outperform state-of-the-art hand-crafted architectures across major face recognition benchmarks.

The TNAS Framework

Search Space and Differentiable Optimization

TNAS treats architecture design as a differentiable optimization problem. The authors define a flexible search space where the number of channels (width) and the number of layers (depth) are treated as variables. To navigate this massive space efficiently, they employ a "super-net" approach combined with the Gumbel-Softmax trick. This allows the choice of architectural parameters to be differentiable, enabling the use of standard gradient descent to "learn" the optimal structure alongside the network weights.

Feature Knowledge Distillation (FKD)

While many KD methods focus on mimicking class probabilities (soft labels), face recognition relies on the geometry of the feature space. TNAS utilizes Feature Fitting, forcing the student’s internal feature vectors to align with those of the teacher. This approach, termed Feature Knowledge Distillation (FKD), is more effective for metric learning tasks where the model must generalize to identities not present in the training set.

Experimental Evaluation

Datasets and Setup

The authors conducted a rigorous evaluation using cleaned versions of popular datasets to ensure fair testing. Training was performed on CASIA-WebFace-R (0.39M images) and MS-Celeb-1M-v1c-R (3.28M images), with overlapping identities removed. Testing spanned nine benchmarks, including LFW, MegaFace, and Trillion-Pairs.

Performance and Efficiency

The results demonstrate that TNAS-searched models consistently outperform standard architectures of similar computational complexity. For instance, on the challenging MegaFace identification task, TNAS significantly exceeds the performance of standard training and other distillation methods like FitNet.

Identification rate comparison with 1M distractors, showing TNAS outperforming other distillation methods.

As shown in the table below, TNAS-D achieves superior accuracy compared to a standard ResNet-50 (R50) while maintaining a similar FLOP count.

Performance comparison of TNAS-D against standard ResNet50 (R50) and the teacher model (SER100).

Key insights from the experiments include:

  • Simultaneous Search: Searching for width and depth together is more effective than optimizing either in isolation.
  • Feature Superiority: Mimicking teacher features (FKD) is far more powerful than mimicking class probabilities (PKD) for face recognition.
  • Generalization: TNAS models showed improved "fairness" on the RFW dataset, suggesting that automated architecture search leads to better generalization across diverse populations.

Conclusion

TNAS represents a significant step toward automating the design of efficient face recognition models. By moving away from hand-crafted heuristics and allowing the teacher model to guide the architectural search, TNAS achieves a superior balance between accuracy and computational cost. Future work may expand this search to include more diverse macro-architectures and explore co-evolutionary strategies between teachers and students.

Summary updated

BibTeX
@inproceedings{wang2021teacher,
  author = {Xiaobo Wang},
  title = {{Teacher Guided Neural Architecture Search for Face Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16387},
}
Figure