Trade-offs in Cross-Domain Generalization of Foundation Model Fine-Tuned for Biometric Applications

Tahar Chettaoui, Naser Damer, Fadi BoutrosarXiv 2025 · arXiv

TL;DR

Fine-tuning foundation models like CLIP for specialized biometric tasks significantly boosts domain-specific performance but triggers catastrophic forgetting of general vision capabilities. This paper systematically quantifies these trade-offs, revealing that complex multi-class tasks (like face recognition) degrade generalization far more than binary tasks (like attack detection), though larger model architectures can partially buffer against this loss.

What Problem Does This Paper Address?

Vision foundation models, most notably CLIP, are celebrated for their remarkable zero-shot generalization across diverse tasks. In biometrics—encompassing face recognition (FR), morphing attack detection (MAD), and presentation attack detection (PAD)—data collection is heavily constrained by privacy laws, ethics, and consent. Consequently, adapting pre-trained foundation models via parameter-efficient methods has become a highly attractive alternative to training specialized models from scratch.

However, a critical trade-off emerges: fine-tuning often leads to "over-specialization" or "catastrophic forgetting." While the model excels at its new biometric task, it may lose its foundational strength: cross-domain generalization. This paper addresses a key question: To what degree do foundation models lose their cross-domain and cross-task generalization capabilities when fine-tuned for specific biometric applications?

Methods

The authors systematically evaluate three recently published, publicly available biometric models that adapt the CLIP image encoder using Low-Rank Adaptation (LoRA):

  1. FRoundation: Fine-tuned for face recognition using a multi-class margin-penalty softmax loss (CosFace).
  2. MADation: Adapted for morphing attack detection via a binary classification head.
  3. FoundPAD: Adapted for presentation attack detection (anti-spoofing) using a task-specific binary classification head.

To quantify the loss of generalization, the authors evaluate these adapted models alongside the original un-tuned CLIP baseline using both zero-shot classification and linear-probe evaluation (where the image encoder acts as a fixed feature extractor for a logistic regression classifier). They test these dynamics across both ViT-B/16 and ViT-L/14 architectures.

Datasets

The evaluation spans both general computer vision and domain-specific biometric benchmarks:

  • General Vision Benchmarks (14 datasets): Used to assess cross-task generalization. This includes ImageNet-v2, CIFAR-10/100, MNIST, Rendered SST2 (OCR), Country211 (geolocation), DTD (textures), SUN397 (scene recognition), and several fine-grained classification datasets (FGVC Aircraft, Stanford Cars, Flowers102, Food101, STL-10, GTSRB).
  • Biometric Benchmarks:
  • Face Recognition: LFW, CFP-FP, AgeDB-30, CALFW, CPLFW, IJB-B, and IJB-C.
  • Morphing Attack Detection: MAD22 and MorDIFF benchmarks.
  • Presentation Attack Detection: Models trained on OULU-NPU, CASIA-FASD, and MSU-MFSD, and evaluated on the large-scale CelebA-Spoof dataset.
Summary of general datasets used for evaluation
Overview of biometric training and evaluation datasets

Key Insights and Ideas

  • Task Complexity Drives Forgetting: Multi-class fine-tuning (FRoundation, trained on 200K identities) causes drastically more catastrophic forgetting than binary classification tasks (MADation and FoundPAD). Optimizing a model for fine-grained identity discrimination strips away its broad semantic understanding.
  • Model Capacity as a Buffer: Larger architectures (ViT-L/14) retain significantly more of their original generalization capability compared to smaller ones (ViT-B/16) after fine-tuning. Increased model capacity acts as a shield against over-specialization.
  • The OCR Anomaly: In zero-shot settings, FoundPAD actually slightly outperformed baseline CLIP on OCR-related tasks (MNIST, SST2). This highlights CLIP's known high variability on text-in-image tasks and suggests binary tuning may occasionally stabilize certain features.

Key Results

  • Biometric Performance Gains: Fine-tuning delivers massive boosts on target tasks. On the large-scale IJB-C face recognition benchmark, FRoundation (ViT-L) achieved a True Acceptance Rate (TAR) of 93.40% (at FAR=1e-4), compared to baseline CLIP's 44.69%—an improvement of up to 58.52%.
  • The Cost of Specialization: This biometric excellence comes at a steep cost for general tasks. On ImageNet-v2, FRoundation's zero-shot accuracy plummeted to 51.63% compared to baseline CLIP's 69.84%.
  • Linear Probing vs. Zero-Shot: Linear probing recovered some general performance across the board, but the performance gap between the specialized FR model and baseline CLIP remained stark. Conversely, binary models (MADation, FoundPAD) maintained near-baseline general performance, with Generalization Loss (GELO) scores close to 1.0.
Table 4. Evaluation of linear-probe cross-task generalization across diverse benchmarks, discussed in Section 4, presented as the classifi- cation accuracy (%). While zero-shot performance shows a correlation with linear-probe accuracy, it remains largely sub-optimal.
Biometric benchmark performance comparison

Limitations and Future Work

The study focuses exclusively on CLIP and the Vision Transformer (ViT) architecture; other foundation models (like DINOv2 or SAM) or CNN backbones were not explored. Additionally, the text encoder of CLIP was kept frozen during fine-tuning across all approaches, which might limit the zero-shot alignment capabilities of the adapted models.

The authors suggest that future work should focus on developing adaptation techniques that strike a better balance—preserving broad generalization while still achieving high task-specific accuracy, potentially through regularized fine-tuning or selective parameter freezing.

Summary updated

Abstract

Foundation models such as CLIP have demonstrated exceptional zero- and few-shot transfer capabilities across diverse vision tasks. However, when fine-tuned for highly specialized biometric tasks, face recognition (FR), morphing attack detection (MAD), and presentation attack detection (PAD), these models may suffer from over-specialization. Thus, they may lose one of their foundational strengths, cross-domain generalization. In this work, we systematically quantify these trade-offs by evaluating three instances of CLIP fine-tuned for FR, MAD, and PAD. We evaluate each adapted model as well as the original CLIP baseline on 14 general vision datasets under zero-shot and linear-probe protocols, alongside common FR, MAD, and PAD benchmarks. Our results indicate that fine-tuned models suffer from over-specialization, especially when fine-tuned for complex tasks of FR. Also, our results pointed out that task complexity and classification head design, multi-class (FR) vs. binary (MAD and PAD), correlate with the degree of catastrophic forgetting. The FRoundation model with the ViT-L backbone outperforms other approaches on the large-scale FR benchmark IJB-C, achieving an improvement of up to 58.52%. However, it experiences a substantial performance drop on ImageNetV2, reaching only 51.63% compared to 69.84% achieved by the baseline CLIP model. Moreover, the larger CLIP architecture consistently preserves more of the model's original generalization ability than the smaller variant, indicating that increased model capacity may help mitigate over-specialization.
BibTeX
@misc{chettaoui2025trade,
  author = {Tahar Chettaoui and Naser Damer and Fadi Boutros},
  title = {{Trade-offs in Cross-Domain Generalization of Foundation Model Fine-Tuned for Biometric Applications}},
  year = {2025},
  eprint = {2509.14921},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.14921},
}
Figure