Abstract
Fine-tuning large-scale pretrained models is a standard paradigm in computer vision. However, standard fine-tuning often suffers from feature distortion and requires significant target-specific data. We present StochCA, a novel fine-tuning strategy for Vision Transformers (ViTs) that stochastically replaces standard self-attention with cross-attention to a frozen, pretrained reference model. This mechanism allows the target model to selectively query pretrained knowledge during training without incurring additional computational costs during inference. Our results demonstrate significant improvements in transfer learning and domain generalization, particularly in low-data regimes.
Introduction
The advent of foundation models pretrained on massive datasets like ImageNet or CLIP has revolutionized computer vision. The typical workflow involves fine-tuning these models on specific downstream tasks. Despite its success, "naïve" fine-tuning—updating all weights—presents two significant challenges:
- Data Efficiency: Achieving peak performance often necessitates large volumes of task-specific data.
- Feature Distortion: The optimization process can degrade the general-purpose representations learned during pretraining, reducing the model's robustness to distribution shifts.
StochCA addresses these issues by treating the pretrained model as a stable "key-value store" of knowledge. Rather than just using pretrained weights as an initialization, StochCA enables the target model to actively consult the pretrained expert throughout the learning process.
Methodology: Stochastic Cross-Attention
The core of our approach is the Stochastic Cross-Attention (StochCA) mechanism. In a standard Vision Transformer (ViT), each layer performs self-attention on its own features. StochCA introduces a stochastic choice at each attention layer during training.
The Mechanism
Given a target model $f$ and a frozen pretrained model $f_0$ with identical architectures, the attention operation in the $i$-th layer is determined by a probability $p$:
- Self-Attention (SA): With probability $1-p$, the model performs standard self-attention using its own queries ($Q_f$), keys ($K_f$), and values ($V_f$).
- Cross-Attention (CA): With probability $p$, the model performs cross-attention. It uses queries from the target model ($Q_f$) but retrieves keys ($K_{f_0}$) and values ($V_{f_0}$) from the corresponding layer of the frozen pretrained model.
This stochastic switching forces the target model's query projections and subsequent MLP layers to remain compatible with both its evolving task-specific features and the high-quality general features of the pretrained expert.
Architecture and Zero-Cost Inference
A key advantage of StochCA is its efficiency. During training, the model shares Layer Norm and MLP weights across both attention paths, ensuring the entire block learns from the pretrained reference.
During inference, the probability $p$ is set to 0, effectively reverting the model to a standard ViT architecture. The pretrained reference model is discarded, ensuring that StochCA provides the benefits of expert consultation during training without any memory or latency overhead at runtime.
Experimental Evaluation
We evaluated StochCA across eight image classification benchmarks focusing on two key areas:
- Transfer Learning (TL): Fine-grained classification tasks including CUB-200-2011, Stanford Cars, FGVC-Aircraft, and Stanford Dogs.
- Domain Generalization (DG): Robustness tests across diverse styles (Photos, Art, Cartoon, Sketch) using PACS, VLCS, OfficeHome, and DomainNet.
Experiments utilized ViT-Base/16 backbones pretrained on ImageNet-1K and CLIP.
Results and Discussion
StochCA demonstrates superior performance, particularly when training data is scarce. On the CUB dataset with only 15% of the training data, StochCA achieved 60.14% accuracy, a significant improvement over the 56.09% achieved by standard fine-tuning.
In Domain Generalization tasks, StochCA consistently outperformed state-of-the-art methods. Using a CLIP-pretrained backbone, it reached an average accuracy of 79.3% across DG benchmarks. Furthermore, StochCA exhibits strong synergy with other fine-tuning techniques; when combined with MIRO, performance reached 81.1%, representing a 13.6% improvement over the Empirical Risk Minimization (ERM) baseline.
Analysis of feature similarity confirms that StochCA-trained models maintain higher cosine similarity to the original pretrained features than those trained via standard fine-tuning. This suggests that the stochastic cross-attention mechanism effectively mitigates feature distortion.
Conclusion and Future Work
StochCA offers a simple yet effective strategy for leveraging pretrained knowledge during fine-tuning. By stochastically consulting a frozen reference model, it improves data efficiency and robustness without increasing inference costs. Future research will explore the application of StochCA to dense prediction tasks like object detection and its potential integration with traditional knowledge distillation frameworks.
Summary updated