TL;DR
Vision Transformers (ViTs) achieve state-of-the-art face recognition accuracy but are too computationally heavy for edge devices. This paper introduces ViT-FREE, a training-free framework that enables early exiting from intermediate layers, achieving up to a 20% inference speedup with negligible accuracy loss. The authors also propose ViT-FREE_FT, a lightweight strategy that uses a small synthetic dataset to dramatically boost the accuracy of early layers without modifying the backbone network.
What Problem Does This Paper Address?
Vision Transformers have become highly popular for Face Recognition (FR) due to their ability to capture global contextual relationships. However, their heavy computational footprint—largely driven by the global self-attention mechanism—makes them difficult to deploy on resource-constrained edge and mobile devices.
While common efficiency strategies like token pruning or architectural redesigns work well for general vision tasks, they aren't optimized for the specific demands of face verification. Furthermore, standard ViT-based face recognition models rely exclusively on the final layer's representation, completely ignoring the highly discriminative features that are already formed in intermediate layers.
Methods
The authors exploit a unique architectural property of ViTs: unlike CNNs, which change feature dimensions across blocks, ViTs maintain a constant token dimensionality throughout all transformer layers. This allows for two novel approaches:
- ViT-FREE (Early Exiting): A completely training-free multi-exit framework. Because of the uniform feature space, the pre-trained final projection head can be directly attached to any intermediate layer to extract face embeddings. This allows the model to terminate inference early, saving compute.
- ViT-FREE_FT (Synthetic Adaptation): Early layers are not inherently optimized for identity discrimination. To fix this, the authors introduce exit-specific projection heads and fine-tune them independently using a margin-based softmax loss (CosFace). Crucially, the entire transformer backbone remains frozen; only the lightweight linear projection layers are updated.
Datasets
- Pre-training: The backbone ViT model was trained on MS1MV2, a refined version of the MS-Celeb-1M dataset containing 5.8M images across 85K identities.
- Fine-tuning: IDPertrub, a compact synthetic dataset consisting of 0.5M images of 10K identities, was used exclusively to tune the early exit projection heads.
- Evaluation: The approach was rigorously tested across multiple standard FR benchmarks: LFW, CFP-FP, AgeDB30, CALFW, CPLFW, IJB-B, IJB-C, and the unconstrained, low-resolution TinyFace benchmark.
Key Insights and Ideas
- Incremental Feature Refinement: By analyzing cosine similarity, the authors found that features change very gradually from layer to layer (with consecutive layer similarity often exceeding 0.90). This proves that ViTs perform incremental refinement, meaning intermediate layers are stable enough to support early exiting.
- Attention Stabilization: Visualization reveals that attention maps in early layers are diffuse and center-focused. As data moves deeper, the attention maps progressively lock onto semantically meaningful facial regions like the eyes, nose, and mouth.
- Synthetic Alignment: You don't need massive real-world datasets to fix early exits. Tuning just the linear projection layer on a small synthetic dataset successfully aligns early-layer features with the final target embedding space.
Key Results
- The Sweet Spot: Exiting at layer 10 (out of 12) provides an optimal trade-off, yielding a 20% inference speedup while incurring only a ~1.5% drop in True Acceptance Rate (TAR) on the challenging IJB-C benchmark (from 96.81% down to 95.36%).
- Massive Shallow Exit Boost: Fine-tuning with synthetic data drastically rescues the performance of early layers. For example, at Exit 7, ViT-FREE_FT pushes the IJB-C TAR from 70.49% up to 88.77% without adding any inference latency or modifying the backbone.
- Cross-Architecture Consistency: The authors validated these findings on a larger 24-layer model (ViT-B KP-RPE), confirming that the progressive refinement behavior holds true across different ViT scales.
Limitations and Future Work
While the synthetic fine-tuning strategy significantly improves shallow exits, it still falls short of matching the representation quality achieved by full-depth pre-training on massive, curated real-world datasets.
Additionally, this work evaluates fixed, static exit depths across the entire dataset. The authors note that a natural next step is implementing an adaptive, per-sample dynamic routing policy—allowing the network to automatically exit early for "easy," high-quality face images and compute deeper layers only for complex, degraded inputs.
Summary updated