TL;DR
Fine-tuning foundation models like CLIP for specialized biometric tasks significantly boosts domain-specific performance but triggers catastrophic forgetting of general vision capabilities. This paper systematically quantifies these trade-offs, revealing that complex multi-class tasks (like face recognition) degrade generalization far more than binary tasks (like attack detection), though larger model architectures can partially buffer against this loss.
What Problem Does This Paper Address?
Vision foundation models, most notably CLIP, are celebrated for their remarkable zero-shot generalization across diverse tasks. In biometrics—encompassing face recognition (FR), morphing attack detection (MAD), and presentation attack detection (PAD)—data collection is heavily constrained by privacy laws, ethics, and consent. Consequently, adapting pre-trained foundation models via parameter-efficient methods has become a highly attractive alternative to training specialized models from scratch.
However, a critical trade-off emerges: fine-tuning often leads to "over-specialization" or "catastrophic forgetting." While the model excels at its new biometric task, it may lose its foundational strength: cross-domain generalization. This paper addresses a key question: To what degree do foundation models lose their cross-domain and cross-task generalization capabilities when fine-tuned for specific biometric applications?
Methods
The authors systematically evaluate three recently published, publicly available biometric models that adapt the CLIP image encoder using Low-Rank Adaptation (LoRA):
- FRoundation: Fine-tuned for face recognition using a multi-class margin-penalty softmax loss (CosFace).
- MADation: Adapted for morphing attack detection via a binary classification head.
- FoundPAD: Adapted for presentation attack detection (anti-spoofing) using a task-specific binary classification head.
To quantify the loss of generalization, the authors evaluate these adapted models alongside the original un-tuned CLIP baseline using both zero-shot classification and linear-probe evaluation (where the image encoder acts as a fixed feature extractor for a logistic regression classifier). They test these dynamics across both ViT-B/16 and ViT-L/14 architectures.
Datasets
The evaluation spans both general computer vision and domain-specific biometric benchmarks:
- General Vision Benchmarks (14 datasets): Used to assess cross-task generalization. This includes ImageNet-v2, CIFAR-10/100, MNIST, Rendered SST2 (OCR), Country211 (geolocation), DTD (textures), SUN397 (scene recognition), and several fine-grained classification datasets (FGVC Aircraft, Stanford Cars, Flowers102, Food101, STL-10, GTSRB).
- Biometric Benchmarks:
- Face Recognition: LFW, CFP-FP, AgeDB-30, CALFW, CPLFW, IJB-B, and IJB-C.
- Morphing Attack Detection: MAD22 and MorDIFF benchmarks.
- Presentation Attack Detection: Models trained on OULU-NPU, CASIA-FASD, and MSU-MFSD, and evaluated on the large-scale CelebA-Spoof dataset.
Key Insights and Ideas
- Task Complexity Drives Forgetting: Multi-class fine-tuning (FRoundation, trained on 200K identities) causes drastically more catastrophic forgetting than binary classification tasks (MADation and FoundPAD). Optimizing a model for fine-grained identity discrimination strips away its broad semantic understanding.
- Model Capacity as a Buffer: Larger architectures (ViT-L/14) retain significantly more of their original generalization capability compared to smaller ones (ViT-B/16) after fine-tuning. Increased model capacity acts as a shield against over-specialization.
- The OCR Anomaly: In zero-shot settings, FoundPAD actually slightly outperformed baseline CLIP on OCR-related tasks (MNIST, SST2). This highlights CLIP's known high variability on text-in-image tasks and suggests binary tuning may occasionally stabilize certain features.
Key Results
- Biometric Performance Gains: Fine-tuning delivers massive boosts on target tasks. On the large-scale IJB-C face recognition benchmark, FRoundation (ViT-L) achieved a True Acceptance Rate (TAR) of 93.40% (at FAR=1e-4), compared to baseline CLIP's 44.69%—an improvement of up to 58.52%.
- The Cost of Specialization: This biometric excellence comes at a steep cost for general tasks. On ImageNet-v2, FRoundation's zero-shot accuracy plummeted to 51.63% compared to baseline CLIP's 69.84%.
- Linear Probing vs. Zero-Shot: Linear probing recovered some general performance across the board, but the performance gap between the specialized FR model and baseline CLIP remained stark. Conversely, binary models (MADation, FoundPAD) maintained near-baseline general performance, with Generalization Loss (GELO) scores close to 1.0.
Limitations and Future Work
The study focuses exclusively on CLIP and the Vision Transformer (ViT) architecture; other foundation models (like DINOv2 or SAM) or CNN backbones were not explored. Additionally, the text encoder of CLIP was kept frozen during fine-tuning across all approaches, which might limit the zero-shot alignment capabilities of the adapted models.
The authors suggest that future work should focus on developing adaptation techniques that strike a better balance—preserving broad generalization while still achieving high task-specific accuracy, potentially through regularized fine-tuning or selective parameter freezing.
Summary updated