Introduction
Fine-tuning pre-trained Vision Transformers (ViTs) on downstream tasks is a standard practice in computer vision. However, this process often triggers "catastrophic forgetting," where the model loses the foundational knowledge acquired during self-supervised pre-training. This research investigates how Parameter-Efficient Fine-tuning (PEFT) techniques can mitigate this stability-plasticity dilemma, allowing models to adapt to new domains while preserving original capabilities.
Methodology
The study compares traditional fine-tuning approaches with two prominent PEFT strategies: Block Expansion and Low-Rank Adaptation (LoRA).
- Standard Fine-tuning: Includes updating all parameters (Full) or only specific subsets like the final layers (Top-k) or the classification head (Linear Probing).
- Block Expansion: This method introduces new "identity blocks" initialized to perform identity mapping. During training, original blocks are frozen, and only these new blocks are updated.
- Low-Rank Adaptation (LoRA): Trainable low-rank matrices are injected into the self-attention mechanism, allowing task-specific adaptation with minimal parameter updates.
Experimental Results
Models pre-trained on ImageNet-1K using DINO were fine-tuned on diverse datasets including Food101, Flowers102, DTD, and CIFAR. Performance was measured by target task accuracy and source domain (ImageNet-1K) retention using k-Nearest Neighbors (K-NN).
Key findings include:
- Stability: Block Expansion emerged as the most robust technique. While full fine-tuning on CIFAR-100 caused ImageNet-1K accuracy to plummet to 25.24%, Block Expansion maintained it at 74.61%.
- Performance: Across five transfer datasets, Block Expansion (p=3) achieved a mean accuracy of 81.60%, significantly outperforming full fine-tuning (56.69%) and LoRA (~77%).
- Learning Rates: Higher learning rates accelerate adaptation but worsen forgetting. Block Expansion at lower rates provides the optimal balance.
Discussion and Conclusion
PEFT serves as a structural shield against forgetting. By freezing the backbone and only training small auxiliary components, foundational features remain intact. While Block Expansion increases the total parameter count compared to LoRA, its superior stability makes it a preferred choice for maintaining general-purpose features in dynamic learning environments. Future work will focus on scaling these methods to larger foundation models and complex continual learning sequences.
Summary updated