Zero-Shot Embeddings Inform Learning and Forgetting with Vision-Language Encoders

Laura Niss, Kevin Vogt-Lowell, Theodoros Tsiligkaridis

Abstract

Despite the proliferation of large vision-language foundation models, estimation of the learning and forgetting outcomes following fine-tuning of these models remains largely unexplored. Inspired by work highlighting the significance of the modality gap in contrastive dual-encoders, we propose the Inter-Intra Modal Measure (IIMM). Combining terms quantifying the similarity between image embeddings and the similarity between incorrect image and label embedding pairs, the IIMM functions as a strong predictor of performance changes with fine-tuning. Our extensive empirical analysis across four state-of-the-art vision-language models (CLIP, SigLIP, CoCa, EVA-02-CLIP) and five fine-tuning techniques (full fine-tuning, BitFit, attention-weight tuning, LoRA, CLIP-Adapter) demonstrates a strong, statistically significant linear relationship: fine-tuning on tasks with higher IIMM scores produces greater in-domain performance gains but also induces more severe out-of-domain performance degradation, with some parameter-efficient fine-tuning (PEFT) methods showing extreme forgetting. We compare our measure against transfer scores from state-of-the-art model selection methods and show that the IIMM is significantly more predictive of accuracy gains. With only a single forward pass of the target data, practitioners can leverage this key insight to heuristically evaluate the degree to which a model can be expected to improve following fine-tuning. Given additional knowledge about the model's performance on a few diverse tasks, this heuristic further evolves into a strong predictor of expected performance changes when training for new tasks.

Introduction

Fine-tuning large vision-language foundation models like CLIP has become a standard practice for adapting general-purpose encoders to specialized domains, such as medical imaging or satellite classification. However, this process is often a double-edged sword. While it can significantly boost in-domain performance, it is computationally expensive and frequently leads to "catastrophic forgetting," where the model loses its broad, zero-shot capabilities.

Until now, practitioners had few tools to estimate whether fine-tuning would be successful without actually performing the training. Existing transferability metrics often require labeled source data or are limited to single-modality models. This paper introduces the Inter-Intra Modal Measure (IIMM), a lightweight metric designed specifically for dual-encoder models that predicts both the potential for learning and the risk of forgetting using only a single forward pass of the target data.

The Inter-Intra Modal Measure (IIMM)

The IIMM is derived directly from the principles of contrastive learning (specifically the InfoNCE loss). It combines two geometric features of the zero-shot embedding space:

  1. Inter-modal Misalignment ($S_{inter}$): This measures the "overlap" or similarity between image embeddings and incorrect text embeddings. High misalignment suggests the model hasn't yet clearly separated distinct concepts across modalities for the target task.
  2. Intra-modal Uniformity ($S_{intra}$): This measures how tightly clustered the image embeddings are to each other. High clustering suggests the representations are redundant and have "room to grow."
Conceptual overview of how high IIMM leads to larger expected gains through embedding space reorganization.

The core insight, termed the "Room to Grow" Hypothesis, suggests that a "messy" initial embedding space (high IIMM) is actually a sign of high potential. If a model’s zero-shot embeddings are densely clustered and misaligned for a specific task, fine-tuning has a massive opportunity to reorganize that space, leading to substantial accuracy gains. Conversely, tasks with low IIMM—where embeddings are already well-distributed and aligned—offer smaller expected gains because there is less geometric room for improvement.

Experimental Evaluation

The researchers evaluated IIMM across a diverse suite of nine vision classification benchmarks, including general datasets (CIFAR100, MNIST), fine-grained tasks (Stanford Cars, GTSRB), and specialized domains (EuroSAT, RESISC45). They tested these against four state-of-the-art models (CLIP, EVA-02-CLIP, SigLIP, and CoCa) and five different fine-tuning strategies, ranging from full fine-tuning to Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA and BitFit.

Results: Predicting Gains and Forgetting

The primary finding is a remarkably strong and statistically significant linear relationship between IIMM and performance changes. For all base models tested, IIMM achieved an $R^2$ value greater than 0.85 (and as high as 0.92) when predicting in-domain accuracy gains.

Linear relationship between IIMM and performance gains (top) and OOD accuracy change (bottom) across four foundation models.

However, this potential for gain comes with a clear trade-off. High IIMM scores are also strongly correlated with higher rates of catastrophic forgetting. As shown in the bottom row of the figure above, as IIMM increases, the magnitude of out-of-distribution (OOD) accuracy degradation also tends to increase. This suggests that the large-scale movement of embeddings required to solve a high-IIMM task often disrupts the delicate alignments that supported the model's original zero-shot capabilities.

Furthermore, the study reveals that PEFT methods are not a universal cure for forgetting. While methods like LoRA are often intended to preserve general knowledge, they can still suffer from significant performance drops when the initial IIMM is high. This highlights the importance of using IIMM as a diagnostic tool regardless of the chosen fine-tuning strategy.

Conclusion and Future Work

IIMM provides a computationally efficient "predictive lens" for foundation model fine-tuning. By requiring only a single forward pass of the data through frozen encoders, practitioners can accurately estimate the expected improvement and the risk of forgetting before committing to expensive training runs.

Future work may explore whether the IIMM framework extends to other tasks like image retrieval or object detection, and its applicability to more complex fusion-based models or large multi-modal models (LMMs) with autoregressive decoders.

Summary updated

Abstract

Despite the proliferation of large vision-language foundation models, estimation of the learning and forgetting outcomes following fine-tuning of these models remains largely unexplored. Inspired by work highlighting the significance of the modality gap in contrastive dual-encoders, we propose the Inter-Intra Modal Measure (IIMM). Combining terms quantifying the similarity between image embeddings and the similarity between incorrect image and label embedding pairs, the IIMM functions as a strong predictor of performance changes with fine-tuning. Our extensive empirical analysis across four state-of-the-art vision-language models (CLIP, SigLIP, CoCa, EVA-02-CLIP) and five fine-tuning techniques (full fine-tuning, BitFit, attention-weight tuning, LoRA, CLIP-Adapter) demonstrates a strong, statistically significant linear relationship: fine-tuning on tasks with higher IIMM scores produces greater in-domain performance gains but also induces more severe out-of-domain performance degradation, with some parameter-efficient fine-tuning (PEFT) methods showing extreme forgetting. We compare our measure against transfer scores from state-of-the-art model selection methods and show that the IIMM is significantly more predictive of accuracy gains. With only a single forward pass of the target data, practitioners can leverage this key insight to heuristically evaluate the degree to which a model can be expected to improve following fine-tuning. Given additional knowledge about the model's performance on a few diverse tasks, this heuristic further evolves into a strong predictor of expected performance changes when training for new tasks.
BibTeX
@misc{niss2024zero,
  author = {Laura Niss and Kevin Vogt-Lowell and Theodoros Tsiligkaridis},
  title = {{Zero-Shot Embeddings Inform Learning and Forgetting with Vision-Language Encoders}},
  year = {2024},
  eprint = {2407.15731},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.15731},
}
Figure