TL;DR
This paper tackles the challenge of transferring knowledge from powerful Transformer-based face recognition models to efficient CNNs, a process hindered by architectural differences. They introduce a novel "Unified Receptive Fields Mapping" module to align feature representations and an "Adaptable Prompting Teacher" to better guide the student, significantly improving performance on mobile-friendly face recognition.
What Problem Does This Paper Address?
Face recognition has seen tremendous progress, but deploying these models on resource-constrained devices like smartphones remains a challenge. While Transformers offer superior performance, their computational demands and lack of platform acceleration make them unsuitable for mobile. Convolutional Neural Networks (CNNs), on the other hand, are well-optimized for hardware acceleration. The natural solution is to distill knowledge from a Transformer "teacher" to a CNN "student." However, existing knowledge distillation (KD) methods primarily focus on transferring knowledge between similar architectures (CNN to CNN). This paper addresses the specific problem of Cross-Architecture Knowledge Distillation (CAKD) from Transformers to CNNs for face recognition, which suffers from two main issues:
- Disparate Spatial Information: Transformers and CNNs process spatial information differently due to their distinct architectural components (e.g., shifted window attention in Transformers vs. standard convolutions in CNNs). This leads to an "unalignment" of receptive fields, making it difficult to align feature spaces.
- Teacher's Inexperience: Traditional KD assumes the teacher is already proficient in its task. However, when a Transformer is used as a teacher for a CNN student, it's not inherently trained to guide a different architecture, lacking the specific "knowledge" needed for effective distillation.
Methods
The authors propose two key innovations to overcome these challenges:
- Unified Receptive Fields Mapping (URFM): This module aims to synchronize the spatial information between teacher and student features. It achieves this by:
- Using learnable "local centers" as query embeddings.
- Supplementing these with "facial positional encoding" that considers facial key points.
- The goal is to map pixel-wise features from both architectures into local features with congruent receptive fields, effectively bridging the architectural gap in spatial representation.
- Adaptable Prompting Teacher (APT): This approach addresses the teacher's lack of distillation-specific knowledge. It involves:
- Prompting: Injecting learnable "prompts" into each Transformer basic layer. These prompts act as instructions, guiding the teacher to manage distillation-specific knowledge.
- Adaptable Capacity: The number of these prompts can be adjusted, allowing for control over the teacher's "adaptable capacity." The paper explores the trade-off between the teacher's adaptable capacity and its discriminative capacity, finding that a balanced approach yields the best results for the student. Crucially, during distillation, only these prompts are optimized, while the rest of the teacher network remains frozen.
These modules are integrated into a framework that also includes attention space and feature space alignment losses, using MSE (Mean Squared Error) for both. The final classification loss is ArcFace.
Datasets
The paper uses standard datasets for training and evaluation in face recognition:
- Training Set: MS1M-V2 (5.8M facial images of 85K individuals).
- Testing Sets:
- Small-scale benchmarks: LFW, CFP-FP, CPLFW, AgeDB, CALFW.
- Large-scale benchmarks: IJB-B, IJB-C (used for TPR@FAR=1e-4 and 1e-5 metrics).
- MegaFace Challenge: Evaluated using the protocol from [7].
Key Insights and Ideas
- Architecture Gap is Crucial: The paper strongly emphasizes that the architectural differences between Transformers and CNNs are a primary bottleneck for effective cross-architecture knowledge distillation, leading to performance degradation compared to same-architecture distillation.
- URFM Solves Spatial Mismatch: Visualizations of Effective Receptive Fields (ERF) clearly show how URFM aligns the spatial coverage of features between the teacher and student, which is critical for feature space alignment.
- Prompts as "Teachers' Aids": The APT module demonstrates that by introducing learnable prompts, the teacher can be guided to effectively transfer knowledge without compromising its own discriminative power. This is a novel application of prompting in the KD context.
- The "Sweet Spot" of Adaptable Capacity: The study reveals a trade-off: a teacher with too much adaptable capacity (learning too much during distillation) can lead to overfitting or ineffective guidance, while a teacher with too little (frozen) might not adapt enough. The optimal performance is achieved with a balanced approach, controlled by the number of prompts.
- Generalization: The method shows good generalization by working with different teacher-student pairs (e.g., Swin-S to MobileFaceNet, ViT-S to IResNet-18).
Key Results
The proposed method, combining URFM and APT, achieves state-of-the-art results on several face recognition benchmarks.
- Superior Performance: The method consistently outperforms existing KD techniques on both small and large-scale datasets. For instance, on CFP-FP, CPLFW, AgeDB-30, and CALFW, the proposed method achieves accuracies of 97.20%, 95.58%, 91.78%, and 97.67% respectively, often surpassing other methods by a significant margin.
- Ablation Studies:
- URFM alone provides a substantial performance boost.
- APT further enhances performance, demonstrating the importance of guiding the teacher.
- The combination of URFM and APT yields the best results.
- Experiments on facial structural distance for positional encoding show that Saliency Distance (SD) is more effective than Relative Distance (RD).
- The optimal number of local centers for URFM was found to be 7x7.
Limitations and Future Work
- Computational Cost: While the goal is to enable efficient CNN students, the Transformer teacher and the proposed modules still add computational overhead. The paper doesn't explicitly detail the inference speedup achieved by the student model after distillation.
- Prompt Optimization: The paper focuses on optimizing prompts for distillation. Future work could explore more sophisticated prompt learning strategies or their application in other cross-architecture transfer learning scenarios.
- Teacher-Student Pair Selection: While generalization is shown, further research could explore optimal teacher-student pairings for specific downstream tasks or hardware constraints.
- Beyond Face Recognition: The core ideas of URFM and APT could potentially be adapted to other domains where cross-architecture knowledge distillation is beneficial.
Summary updated