Quantified Task Misalignment to Inform PEFT: An Exploration of Domain Generalization and Catastrophic Forgetting in CLIP

Laura Niss, Kevin Vogt-Lowell, Theodoros TsiligkaridisarXiv 2024 · OpenReview

Abstract

Foundations models are presented as generalists that often perform well over a myriad of tasks. Fine-tuning these models, even on limited data, provides an additional boost in task-specific performance but often at the cost of their wider generalization, an effect termed catastrophic forgetting. In this paper, we analyze the relation between task difficulty in the CLIP model and the performance of several simple parameter-efficient fine-tuning methods through the lens of domain generalization and catastrophic forgetting. We provide evidence that the silhouette score of the zero-shot image and text embeddings is a better measure of task difficulty than the average cosine similarity of correct image/label embeddings, and discuss observable relationships between task difficulty, fine-tuning method, domain generalization, and catastrophic forgetting. Additionally, the averaged results across tasks and performance measures demonstrate that a simplified method that trains only a subset of attention weights, which we call A-CLIP, yields a balance between domain generalization and catastrophic forgetting.

Introduction

Foundation models like CLIP (Contrastive Language-Image Pre-training) are powerful generalists, but they often require fine-tuning to achieve peak performance on specialized tasks, such as identifying satellite imagery or specific vehicle models. This process introduces the "stability-plasticity" dilemma: aggressive fine-tuning (Full Fine-tuning) can trigger catastrophic forgetting, where the model loses its pre-trained general knowledge. Conversely, overly conservative methods may fail to learn the new task effectively or fail to generalize to slightly different data (Domain Generalization). This research investigates how "task difficulty"—defined by the misalignment between image and text spaces before training—dictates the success of various Parameter-Efficient Fine-Tuning (PEFT) methods.

Quantifying Task Difficulty: The Silhouette Score

A key insight of this work is that the Silhouette Score (SS) of embeddings is a superior measure of task difficulty compared to standard average cosine similarity. The silhouette score measures how well-separated different classes are in the embedding space. A high zero-shot SS indicates that the model must significantly shift its embeddings to align the new data, serving as a warning sign for potential catastrophic forgetting.

Figure 6: Analysis of Silhouette Score (ss) vs. Cosine Similarity as measures of task difficulty and their correlation with accuracy and change during fine-tuning.

As shown in Figure 6, the relationship between the zero-shot silhouette score and the change in alignment during training was statistically significant ($p=0.001$), whereas the standard cosine similarity measure was not ($p=0.071$).

Parameter-Efficient Fine-Tuning Methods

The authors compared several PEFT methods that update only a fraction of the model's weights:

  • LoRA: Learns low-rank update matrices.
  • BitFit: Updates only the bias terms.
  • CLIP-Adapter: Adds small bottleneck layers.
  • A-CLIP (Proposed): A "surgical" approach that freezes everything except the in-projection weights of the attention layers, targeting the weights that change the most during standard fine-tuning.

Key Results and Discussion

The study utilized five diverse datasets to evaluate Domain Generalization (DG) and Catastrophic Forgetting (CF).

Figure 1: In-domain and cross-dataset accuracy differences from Zero-Shot (ZS) to Fine-Tuning (FT) across various datasets and PEFT methods.

A-CLIP: The "Goldilocks" Method

A-CLIP achieved a superior balance between task performance and generalization. On the FMoW-ID (satellite) task, A-CLIP reached 58.67% accuracy, outperforming Full Fine-tuning (57.30%) and significantly exceeding LoRA (15.24%). It maintained much better generalization and resisted forgetting across benchmarks.

Performance Trade-offs

  • BitFit is Risky for "Hard" Tasks: While efficient, BitFit is highly susceptible to catastrophic forgetting when the target task has a large zero-shot silhouette score. On the SVHN task, BitFit caused a massive drop in general performance (dropping to 16.61% on evaluation sets).
  • LoRA as a Specialist: LoRA was found to be the most "surgical," restricting changes to specific areas of the embedding space. This makes it excellent at preventing forgetting, but it sometimes hampers the model's ability to generalize to new domains.

Conclusion

By quantifying task misalignment through the silhouette score, researchers can better predict the outcomes of fine-tuning. A-CLIP offers a robust middle ground, providing high task-specific accuracy without sacrificing the model's foundational knowledge. Future work will explore selecting the best fine-tuning strategy automatically based on zero-shot difficulty measures before training begins.

Summary updated

Abstract

Foundations models are presented as generalists that often perform well over a myriad of tasks. Fine-tuning these models, even on limited data, provides an additional boost in task-specific performance but often at the cost of their wider generalization, an effect termed catastrophic forgetting. In this paper, we analyze the relation between task difficulty in the CLIP model and the performance of several simple parameter-efficient fine-tuning methods through the lens of domain generalization and catastrophic forgetting. We provide evidence that the silhouette score of the zero-shot image and text embeddings is a better measure of task difficulty than the average cosine similarity of correct image/label embeddings, and discuss observable relationships between task difficulty, fine-tuning method, domain generalization, and catastrophic forgetting. Additionally, the averaged results across tasks and performance measures demonstrate that a simplified method that trains only a subset of attention weights, which we call A-CLIP, yields a balance between domain generalization and catastrophic forgetting.
BibTeX
@misc{niss2024quantified,
  author = {Laura Niss and Kevin Vogt-Lowell and Theodoros Tsiligkaridis},
  title = {{Quantified Task Misalignment to Inform PEFT: An Exploration of Domain Generalization and Catastrophic Forgetting in CLIP}},
  year = {2024},
  eprint = {2402.09613},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.09613},
}
Figure