ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation

Tahar Chettaoui, Guray Ozgur, Eduarda Caldeira, Naser Damer, Fadi BoutrosarXiv 2026 · arXiv

TL;DR

Vision Transformers (ViTs) achieve state-of-the-art face recognition accuracy but are too computationally heavy for edge devices. This paper introduces ViT-FREE, a training-free framework that enables early exiting from intermediate layers, achieving up to a 20% inference speedup with negligible accuracy loss. The authors also propose ViT-FREE_FT, a lightweight strategy that uses a small synthetic dataset to dramatically boost the accuracy of early layers without modifying the backbone network.

What Problem Does This Paper Address?

Vision Transformers have become highly popular for Face Recognition (FR) due to their ability to capture global contextual relationships. However, their heavy computational footprint—largely driven by the global self-attention mechanism—makes them difficult to deploy on resource-constrained edge and mobile devices.

While common efficiency strategies like token pruning or architectural redesigns work well for general vision tasks, they aren't optimized for the specific demands of face verification. Furthermore, standard ViT-based face recognition models rely exclusively on the final layer's representation, completely ignoring the highly discriminative features that are already formed in intermediate layers.

Methods

The authors exploit a unique architectural property of ViTs: unlike CNNs, which change feature dimensions across blocks, ViTs maintain a constant token dimensionality throughout all transformer layers. This allows for two novel approaches:

  1. ViT-FREE (Early Exiting): A completely training-free multi-exit framework. Because of the uniform feature space, the pre-trained final projection head can be directly attached to any intermediate layer to extract face embeddings. This allows the model to terminate inference early, saving compute.
  2. ViT-FREE_FT (Synthetic Adaptation): Early layers are not inherently optimized for identity discrimination. To fix this, the authors introduce exit-specific projection heads and fine-tune them independently using a margin-based softmax loss (CosFace). Crucially, the entire transformer backbone remains frozen; only the lightweight linear projection layers are updated.
Fig. 2: ViT-FREE inference pipeline. Intermediate patch em- beddings from each transformer block are projected to fea- ture representations, enabling predictions at different depths and allowing a flexible trade-off between computational effi- ciency and recognition performance. The symbol ⊕denotes the concatenation operator.

Datasets

  • Pre-training: The backbone ViT model was trained on MS1MV2, a refined version of the MS-Celeb-1M dataset containing 5.8M images across 85K identities.
  • Fine-tuning: IDPertrub, a compact synthetic dataset consisting of 0.5M images of 10K identities, was used exclusively to tune the early exit projection heads.
  • Evaluation: The approach was rigorously tested across multiple standard FR benchmarks: LFW, CFP-FP, AgeDB30, CALFW, CPLFW, IJB-B, IJB-C, and the unconstrained, low-resolution TinyFace benchmark.

Key Insights and Ideas

  • Incremental Feature Refinement: By analyzing cosine similarity, the authors found that features change very gradually from layer to layer (with consecutive layer similarity often exceeding 0.90). This proves that ViTs perform incremental refinement, meaning intermediate layers are stable enough to support early exiting.
  • Attention Stabilization: Visualization reveals that attention maps in early layers are diffuse and center-focused. As data moves deeper, the attention maps progressively lock onto semantically meaningful facial regions like the eyes, nose, and mouth.
Fig. 5: Attention maps of ViT-FREE at depths 1–12, cor- responding to the attention patterns at intermediate early- exit depths. In the early layers, attention is spread out and mainly concentrated near the center of the image. As the network progresses to deeper layers, attention maps exhibit more defined patterns, with stronger responses around the eyes, nose, and mouth.
  • Synthetic Alignment: You don't need massive real-world datasets to fix early exits. Tuning just the linear projection layer on a small synthetic dataset successfully aligns early-layer features with the final target embedding space.

Key Results

  • The Sweet Spot: Exiting at layer 10 (out of 12) provides an optimal trade-off, yielding a 20% inference speedup while incurring only a ~1.5% drop in True Acceptance Rate (TAR) on the challenging IJB-C benchmark (from 96.81% down to 95.36%).
  • Massive Shallow Exit Boost: Fine-tuning with synthetic data drastically rescues the performance of early layers. For example, at Exit 7, ViT-FREE_FT pushes the IJB-C TAR from 70.49% up to 88.77% without adding any inference latency or modifying the backbone.
Fig. 3: Cosine similarity between intermediate feature em- beddings at each ViT depth (0–10) and the final embedding (depth 11) for both a baseline ViT-FREE and its fine-tuned variant ViT-FREEF T , as defined in Section IV. ViT-FREEF T consistently achieves higher or comparable similarity to the final feature embedding across all depths, demonstrating the effectiveness of fine-tuning in accelerating the refinement of feature representations throughout the network.
  • Cross-Architecture Consistency: The authors validated these findings on a larger 24-layer model (ViT-B KP-RPE), confirming that the progressive refinement behavior holds true across different ViT scales.

Limitations and Future Work

While the synthetic fine-tuning strategy significantly improves shallow exits, it still falls short of matching the representation quality achieved by full-depth pre-training on massive, curated real-world datasets.

Additionally, this work evaluates fixed, static exit depths across the entire dataset. The authors note that a natural next step is implementing an adaptive, per-sample dynamic routing policy—allowing the network to automatically exit early for "easy," high-quality face images and compute deeper layers only for complex, degraded inputs.

Summary updated

Abstract

Vision Transformers (ViTs) have gained significant attention in computer vision and shown strong potential for face recognition (FR). However, their high computational cost makes deployment on resource-constrained devices challenging, motivating the need for methods that balance efficiency and accuracy. In this work, we investigate early exiting in pretrained ViTs as a simple yet effective training-free strategy for efficient FR inference. Leveraging the uniform feature dimensionality across transformer encoder blocks, we introduce ViT-FREE, a multi-exit framework that enables face verification directly from intermediate representations without modifying or retraining the backbone model, and thus, reducing inference cost. Empirically, we show that patch embeddings and attention maps evolve progressively across depth, exhibiting high similarity between consecutive ViT blocks and increasing alignment with the final representation. This indicates gradual feature refinement and attention convergence, suggesting that intermediate layers already provide stable and discriminative representations suitable for early exiting. Through extensive experiments on multiple FR benchmarks, we systematically analyze the accuracy-efficiency trade-off across exit depths. Our results demonstrate that later exits achieve a highly favorable balance, with exiting at layer 10 yielding up to a 20% speedup while incurring only a 1.5 drop in verification performance on benchmarks such as IJB-C. Also, we propose ViT-FREE_FT, a lightweight exit-specific fine-tuning strategy that adapts only the projection layers using a small synthetic dataset while keeping the transformer backbone frozen. This approach improves the performance of shallow exits while preserving the efficiency benefits and leaving deeper exits largely unaffected.
BibTeX
@misc{chettaoui2026vit,
  author = {Tahar Chettaoui and Guray Ozgur and Eduarda Caldeira and Naser Damer and Fadi Boutros},
  title = {{ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation}},
  year = {2026},
  eprint = {2606.12023},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.12023},
}

Loading related content...

Figure