Introduction
Fine-tuning large-scale pre-trained models is a standard practice for adapting general-purpose knowledge to specific downstream tasks. However, this process often suffers from "catastrophic forgetting," where the model's performance on its original broad domain degrades as it over-specializes. This paper introduces Vector-based Feature Space Adaptation (VeFA), a novel approach designed to mitigate forgetting by constraining model updates to the original feature space established during pre-training.
The Problem: Intruder Dimensions and Subspace Escape
When fine-tuning models like CLIP or GPT-2, especially in low-data regimes, the model tends to overfit. A primary driver of this phenomenon is the emergence of intruder dimensions—new directions in the model's internal representations that were absent during pre-training.
While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA reduce the number of trainable parameters by operating in the "weight space," they still permit the model's learned representations to "escape" the original subspace. This "subspace escape" is fundamentally linked to catastrophic forgetting, as the model loses the structural constraints that ensured its general-purpose capabilities.
Methodology: Vector-based Feature Space Adaptation (VeFA)
VeFA addresses the issue of subspace escape by operating directly in the feature space. Instead of modifying weights through low-rank matrix additions, VeFA applies an element-wise scaling transformation to individual feature channels.
In a standard neural network layer with pre-trained weights $W_0$, VeFA introduces a diagonal scaling matrix $\Lambda$. The effective update rule is $W_{ft} = W_0 + W_0\Lambda$. Mathematically, this ensures that the fine-tuned weights $W_{ft}$ remain within the column space of the original matrix $W_0$. By construction, VeFA prevents the creation of intruder dimensions. This method draws inspiration from Effect Equivalence Modeling (EEM), a statistical technique used to account for unobserved "lurking variables" without altering the fundamental model parameters.
Experimental Evaluation
The authors evaluated VeFA across diverse domains and architectures:
- Image Classification: Using CLIP (ViT-B/16) on seven datasets including Caltech101, EuroSAT, and Oxford Pets.
- Natural Language Understanding (NLU): Using RoBERTa (Base and Large) on the GLUE benchmark.
- Natural Language Generation (NLG): Using GPT-2 (Medium and Large) on the E2E benchmark.
Key Insights: Subspace Preservation and Efficiency
The core advantage of VeFA lies in its ability to preserve the original representational subspace $S_0$.
As illustrated in the training dynamics, while LoRA's weight updates ($W^{lora}$) tend to move outside the pre-trained subspace $S_0$, VeFA's updates ($W^{vefa}$) are strictly confined within it. This constraint significantly enhances robustness. Furthermore, VeFA is exceptionally parameter-efficient. For instance, in RoBERTa-Base, VeFA utilizes only 0.018M parameters—a 16x reduction compared to LoRA's 0.3M—while maintaining or exceeding performance on out-of-distribution tasks.
Results and Robustness
- Robustness Scores: On the GLUE benchmark with RoBERTa-Base, VeFA achieved an average robustness score of 18.8, significantly higher than LoRA's 4.0.
- Few-Shot Performance: VeFA matched or outperformed LoRA in 1-shot, 4-shot, and 16-shot image classification scenarios across nearly all datasets.
- Generative Tasks: On the E2E benchmark, VeFA demonstrated comparable BLEU and ROUGE-L scores to LoRA, indicating that its constrained update rule still possesses sufficient expressive power for complex generation.
Limitations and Future Outlook
VeFA currently restricts adaptation to channel-wise scaling, which avoids cross-channel mixing. While this prevents intruder dimensions, it may limit the model's ability to learn highly complex new features. Future research could explore "projected gradient schemes" to allow more complex updates within the $S_0$ subspace or investigate sparsity-inducing constraints on the scaling vector to enhance interpretability.
Summary updated