StochCA: A Novel Approach for Exploiting Pretrained Models with Cross-Attention

Seungwon Seo, Suho Lee, Sangheum Hwang

Abstract

Fine-tuning large-scale pretrained models is a standard paradigm in computer vision. However, standard fine-tuning often suffers from feature distortion and requires significant target-specific data. We present StochCA, a novel fine-tuning strategy for Vision Transformers (ViTs) that stochastically replaces standard self-attention with cross-attention to a frozen, pretrained reference model. This mechanism allows the target model to selectively query pretrained knowledge during training without incurring additional computational costs during inference. Our results demonstrate significant improvements in transfer learning and domain generalization, particularly in low-data regimes.

Introduction

The advent of foundation models pretrained on massive datasets like ImageNet or CLIP has revolutionized computer vision. The typical workflow involves fine-tuning these models on specific downstream tasks. Despite its success, "naïve" fine-tuning—updating all weights—presents two significant challenges:

  1. Data Efficiency: Achieving peak performance often necessitates large volumes of task-specific data.
  2. Feature Distortion: The optimization process can degrade the general-purpose representations learned during pretraining, reducing the model's robustness to distribution shifts.

StochCA addresses these issues by treating the pretrained model as a stable "key-value store" of knowledge. Rather than just using pretrained weights as an initialization, StochCA enables the target model to actively consult the pretrained expert throughout the learning process.

Methodology: Stochastic Cross-Attention

The core of our approach is the Stochastic Cross-Attention (StochCA) mechanism. In a standard Vision Transformer (ViT), each layer performs self-attention on its own features. StochCA introduces a stochastic choice at each attention layer during training.

Comparison of Self-Attention (left) and Cross-Attention (right) mechanisms in StochCA. During training, the model stochastically switches between these two modes.

The Mechanism

Given a target model $f$ and a frozen pretrained model $f_0$ with identical architectures, the attention operation in the $i$-th layer is determined by a probability $p$:

  • Self-Attention (SA): With probability $1-p$, the model performs standard self-attention using its own queries ($Q_f$), keys ($K_f$), and values ($V_f$).
  • Cross-Attention (CA): With probability $p$, the model performs cross-attention. It uses queries from the target model ($Q_f$) but retrieves keys ($K_{f_0}$) and values ($V_{f_0}$) from the corresponding layer of the frozen pretrained model.

This stochastic switching forces the target model's query projections and subsequent MLP layers to remain compatible with both its evolving task-specific features and the high-quality general features of the pretrained expert.

Architecture and Zero-Cost Inference

A key advantage of StochCA is its efficiency. During training, the model shares Layer Norm and MLP weights across both attention paths, ensuring the entire block learns from the pretrained reference.

Overview of the StochCA architecture. The left panel illustrates the training phase with shared components, while the right panel shows the streamlined stochastic attention layer.

During inference, the probability $p$ is set to 0, effectively reverting the model to a standard ViT architecture. The pretrained reference model is discarded, ensuring that StochCA provides the benefits of expert consultation during training without any memory or latency overhead at runtime.

Experimental Evaluation

We evaluated StochCA across eight image classification benchmarks focusing on two key areas:

  • Transfer Learning (TL): Fine-grained classification tasks including CUB-200-2011, Stanford Cars, FGVC-Aircraft, and Stanford Dogs.
  • Domain Generalization (DG): Robustness tests across diverse styles (Photos, Art, Cartoon, Sketch) using PACS, VLCS, OfficeHome, and DomainNet.

Experiments utilized ViT-Base/16 backbones pretrained on ImageNet-1K and CLIP.

Results and Discussion

StochCA demonstrates superior performance, particularly when training data is scarce. On the CUB dataset with only 15% of the training data, StochCA achieved 60.14% accuracy, a significant improvement over the 56.09% achieved by standard fine-tuning.

In Domain Generalization tasks, StochCA consistently outperformed state-of-the-art methods. Using a CLIP-pretrained backbone, it reached an average accuracy of 79.3% across DG benchmarks. Furthermore, StochCA exhibits strong synergy with other fine-tuning techniques; when combined with MIRO, performance reached 81.1%, representing a 13.6% improvement over the Empirical Risk Minimization (ERM) baseline.

Analysis of feature similarity confirms that StochCA-trained models maintain higher cosine similarity to the original pretrained features than those trained via standard fine-tuning. This suggests that the stochastic cross-attention mechanism effectively mitigates feature distortion.

Conclusion and Future Work

StochCA offers a simple yet effective strategy for leveraging pretrained knowledge during fine-tuning. By stochastically consulting a frozen reference model, it improves data efficiency and robustness without increasing inference costs. Future research will explore the application of StochCA to dense prediction tasks like object detection and its potential integration with traditional knowledge distillation frameworks.

Summary updated

Abstract

Utilizing large-scale pretrained models is a well-known strategy to enhance performance on various target tasks. It is typically achieved through fine-tuning pretrained models on target tasks. However, na\"{\i}ve fine-tuning may not fully leverage knowledge embedded in pretrained models. In this study, we introduce a novel fine-tuning method, called stochastic cross-attention (StochCA), specific to Transformer architectures. This method modifies the Transformer's self-attention mechanism to selectively utilize knowledge from pretrained models during fine-tuning. Specifically, in each block, instead of self-attention, cross-attention is performed stochastically according to the predefined probability, where keys and values are extracted from the corresponding block of a pretrained model. By doing so, queries and channel-mixing multi-layer perceptron layers of a target model are fine-tuned to target tasks to learn how to effectively exploit rich representations of pretrained models. To verify the effectiveness of StochCA, extensive experiments are conducted on benchmarks in the areas of transfer learning and domain generalization, where the exploitation of pretrained models is critical. Our experimental results show the superiority of StochCA over state-of-the-art approaches in both areas. Furthermore, we demonstrate that StochCA is complementary to existing approaches, i.e., it can be combined with them to further improve performance. Our code is available at https://github.com/daintlab/stochastic_cross_attention
BibTeX
@article{seo2024stochca,
  author = {Seungwon Seo and Suho Lee and Sangheum Hwang},
  title = {{StochCA: A Novel Approach for Exploiting Pretrained Models with Cross-Attention}},
  journal = {Neural Networks},
  year = {2024},
  doi = {10.1016/j.neunet.2024.106663},
}
Figure