Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgetting

Reza Akbarian Bafghi, Nidhin Harilal, Claire Monteleoni, Maziar Raissi

Introduction

Fine-tuning pre-trained Vision Transformers (ViTs) on downstream tasks is a standard practice in computer vision. However, this process often triggers "catastrophic forgetting," where the model loses the foundational knowledge acquired during self-supervised pre-training. This research investigates how Parameter-Efficient Fine-tuning (PEFT) techniques can mitigate this stability-plasticity dilemma, allowing models to adapt to new domains while preserving original capabilities.

Methodology

The study compares traditional fine-tuning approaches with two prominent PEFT strategies: Block Expansion and Low-Rank Adaptation (LoRA).

Comparison of fine-tuning architectures: (a) Linear-probing, (b) Fully fine-tuned, (c) Block Expansion, and (d) LoRA.
  1. Standard Fine-tuning: Includes updating all parameters (Full) or only specific subsets like the final layers (Top-k) or the classification head (Linear Probing).
  2. Block Expansion: This method introduces new "identity blocks" initialized to perform identity mapping. During training, original blocks are frozen, and only these new blocks are updated.
  3. Low-Rank Adaptation (LoRA): Trainable low-rank matrices are injected into the self-attention mechanism, allowing task-specific adaptation with minimal parameter updates.

Experimental Results

Models pre-trained on ImageNet-1K using DINO were fine-tuned on diverse datasets including Food101, Flowers102, DTD, and CIFAR. Performance was measured by target task accuracy and source domain (ImageNet-1K) retention using k-Nearest Neighbors (K-NN).

Accuracy trade-offs on Foods101 and Flowers102. Block Expansion (green star) consistently achieves high accuracy on both target and source domains.

Key findings include:

  • Stability: Block Expansion emerged as the most robust technique. While full fine-tuning on CIFAR-100 caused ImageNet-1K accuracy to plummet to 25.24%, Block Expansion maintained it at 74.61%.
  • Performance: Across five transfer datasets, Block Expansion (p=3) achieved a mean accuracy of 81.60%, significantly outperforming full fine-tuning (56.69%) and LoRA (~77%).
  • Learning Rates: Higher learning rates accelerate adaptation but worsen forgetting. Block Expansion at lower rates provides the optimal balance.

Discussion and Conclusion

PEFT serves as a structural shield against forgetting. By freezing the backbone and only training small auxiliary components, foundational features remain intact. While Block Expansion increases the total parameter count compared to LoRA, its superior stability makes it a preferred choice for maintaining general-purpose features in dynamic learning environments. Future work will focus on scaling these methods to larger foundation models and complex continual learning sequences.

Summary updated

Abstract

Artificial neural networks often suffer from catastrophic forgetting, where learning new concepts leads to a complete loss of previously acquired knowledge. We observe that this issue is particularly magnified in vision transformers (ViTs), where post-pre-training and fine-tuning on new tasks can significantly degrade the model's original general abilities. For instance, a DINO ViT-Base/16 pre-trained on ImageNet-1k loses over 70% accuracy on ImageNet-1k after just 10 iterations of fine-tuning on CIFAR-100. Overcoming this stability-plasticity dilemma is crucial for enabling ViTs to continuously learn and adapt to new domains while preserving their initial knowledge. In this work, we study two new parameter-efficient fine-tuning strategies: (1)~Block Expansion, and (2) Low-rank adaptation (LoRA). Our experiments reveal that using either Block Expansion or LoRA on self-supervised pre-trained ViTs surpass fully fine-tuned ViTs in new domains while offering significantly greater parameter efficiency. Notably, we find that Block Expansion experiences only a minimal performance drop in the pre-training domain, thereby effectively mitigating catastrophic forgetting in pre-trained ViTs.
BibTeX
@inproceedings{bafghi2024parameter,
  author = {Reza Akbarian Bafghi and Nidhin Harilal and Claire Monteleoni and Maziar Raissi},
  title = {{Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgetting}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
  doi = {10.1109/cvprw63382.2024.00371},
}
Figure