Introduction
Knowledge Distillation (KD) has become a cornerstone for deploying deep learning models on resource-constrained devices. By training a compact "student" network to mimic a high-performance "teacher," researchers can achieve significant reductions in model size and latency. However, a critical bottleneck remains: the manual design of the student architecture. Traditionally, researchers rely on hand-crafted heuristics, such as selecting a standard ResNet-18, which may not be optimal for the specific demands of face recognition—an "open-set" problem requiring robust feature embeddings for unseen identities.
This paper introduces Teacher Guided Neural Architecture Search (TNAS), an automated framework that leverages the teacher model's expertise to search for the optimal student architecture. By optimizing network width and depth simultaneously, TNAS discovers compact models that outperform state-of-the-art hand-crafted architectures across major face recognition benchmarks.
The TNAS Framework
Search Space and Differentiable Optimization
TNAS treats architecture design as a differentiable optimization problem. The authors define a flexible search space where the number of channels (width) and the number of layers (depth) are treated as variables. To navigate this massive space efficiently, they employ a "super-net" approach combined with the Gumbel-Softmax trick. This allows the choice of architectural parameters to be differentiable, enabling the use of standard gradient descent to "learn" the optimal structure alongside the network weights.
Feature Knowledge Distillation (FKD)
While many KD methods focus on mimicking class probabilities (soft labels), face recognition relies on the geometry of the feature space. TNAS utilizes Feature Fitting, forcing the student’s internal feature vectors to align with those of the teacher. This approach, termed Feature Knowledge Distillation (FKD), is more effective for metric learning tasks where the model must generalize to identities not present in the training set.
Experimental Evaluation
Datasets and Setup
The authors conducted a rigorous evaluation using cleaned versions of popular datasets to ensure fair testing. Training was performed on CASIA-WebFace-R (0.39M images) and MS-Celeb-1M-v1c-R (3.28M images), with overlapping identities removed. Testing spanned nine benchmarks, including LFW, MegaFace, and Trillion-Pairs.
Performance and Efficiency
The results demonstrate that TNAS-searched models consistently outperform standard architectures of similar computational complexity. For instance, on the challenging MegaFace identification task, TNAS significantly exceeds the performance of standard training and other distillation methods like FitNet.
As shown in the table below, TNAS-D achieves superior accuracy compared to a standard ResNet-50 (R50) while maintaining a similar FLOP count.
Key insights from the experiments include:
- Simultaneous Search: Searching for width and depth together is more effective than optimizing either in isolation.
- Feature Superiority: Mimicking teacher features (FKD) is far more powerful than mimicking class probabilities (PKD) for face recognition.
- Generalization: TNAS models showed improved "fairness" on the RFW dataset, suggesting that automated architecture search leads to better generalization across diverse populations.
Conclusion
TNAS represents a significant step toward automating the design of efficient face recognition models. By moving away from hand-crafted heuristics and allowing the teacher model to guide the architectural search, TNAS achieves a superior balance between accuracy and computational cost. Future work may expand this search to include more diverse macro-architectures and explore co-evolutionary strategies between teachers and students.
Summary updated