Adaptive Attention Link-based Regularization for Vision Transformers

Heegon Jin, Jongwon Choi

TL;DR

Vision Transformers (ViTs) achieve state-of-the-art results but require massive datasets and long training times due to their lack of inductive bias. This paper introduces an "Adaptive Attention Link" (AAL) module that bridges ViTs with pre-trained Convolutional Neural Networks (CNNs) to inject inductive biases, significantly improving ViT training efficiency and performance, especially on smaller datasets.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have emerged as powerful models for computer vision tasks, often surpassing traditional Convolutional Neural Networks (CNNs). However, ViTs lack the inherent "inductive biases" (like locality and translation equivariance) that CNNs possess. This absence means ViTs require enormous amounts of training data and extensive training durations to learn effectively and avoid overfitting. For researchers and practitioners with limited data or computational resources, training ViTs to a competitive level is a significant challenge. This paper aims to bridge this gap by making ViT training more efficient and robust, particularly in low-data regimes.

Methods

The core innovation is the Adaptive Attention Link (AAL) module. This module acts as a regularization technique that connects a pre-trained CNN to a ViT during training. Here's how it works:

  • Attention Augmentation Module: AAL creates trainable "attention links" that learn to combine attention maps from different heads of a ViT. These augmented attention maps are then trained to mimic the activation maps of a pre-trained CNN.
  • Knowledge Transfer: By forcing the ViT's attention maps to align with CNN activation maps, AAL effectively transfers the inductive biases of CNNs to the ViT. This is achieved through an attention-based regularization loss that minimizes the difference between the augmented ViT attention maps and the CNN activation maps.
  • Preservation of ViT Architecture: Crucially, the AAL module is external to the ViT architecture and is only used during training. This means the original ViT architecture remains intact for inference, and the method can be applied to various ViT variants.
  • Selective Attention Links: The authors further propose "selective attention links" (a-link and β-link) which prune less informative connections between CNN and ViT layers. This is based on the observation that ViT attention maps and CNN activation maps have correlations along their respective layer depths, and higher-layer ViT attention heads might capture semantic information beyond CNN capabilities, requiring less regularization.

Datasets

The paper uses the following datasets for training and evaluation:

  • ImageNet: Used extensively for training and evaluation, including full ImageNet and subsampled versions (5%, 10%, 30%, 50%).
  • CIFAR-10: Used for classification tasks.
  • Caltech-UCSD Birds-200-2011 (CUB-200): Used for fine-grained classification and weakly supervised object localization.
  • Oxford 102 Flowers (Flowers-102): Used for fine-grained classification.

Key Insights and Ideas

  • CNN Inductive Bias is Key for ViT Efficiency: The study strongly suggests that injecting CNN's inductive biases (locality) into ViTs is a highly effective way to improve their training efficiency and robustness, especially when data is scarce.
  • Attention Maps as a Bridge: The paper demonstrates that ViT attention maps and CNN activation maps, despite originating from different architectures, exhibit meaningful correlations, particularly at similar layer depths. This correlation is the foundation for the proposed regularization.
  • High-Level ViT Heads Capture Beyond CNNs: Analysis reveals that higher-layer ViT attention heads can learn semantic information that CNNs might not capture. This implies that over-regularizing these heads with CNN biases might be detrimental, leading to the development of selective attention links.
  • Teacher-Student Agnosticism: The AAL method can utilize any pre-trained CNN as a teacher, regardless of whether it was trained on the same dataset as the student ViT. This offers significant flexibility.

Key Results

  • Improved Accuracy with Less Data: On ImageNet with subsampled datasets (e.g., 5% and 10%), the proposed AAL method significantly outperforms baselines like DeiT and ConViT in terms of Top-1 accuracy. For instance, on 5% ImageNet, AAL achieves 56.1% Top-1 accuracy compared to DeiT-S's 34.8% and ConViT-S's 47.8%.
  • Faster Convergence: The paper shows that AAL enables ViTs to reach high accuracy levels much faster. For example, on ImageNet, AAL achieves 70% Top-1 accuracy in about 50 epochs, while DeiT requires 120 epochs.
  • Reduced Overfitting: AAL effectively mitigates overfitting, particularly in low-data regimes, as evidenced by the widening performance gap over baselines when training data is scarce.
  • Competitive Performance with Reduced Parameters: The AAL module adds minimal parameters (0.068M), and the overall model size is comparable to or even smaller than baselines like ConViT, while achieving superior or equivalent performance.
  • Enhanced Localization: In Weakly Supervised Object Localization (WSOL) tasks on CUB-200, AAL shows a substantial improvement in localization accuracy (53.5% higher than DeiT at IoU threshold 0.5), indicating better spatial awareness.
  • Selective Links Further Boost Performance: The selective attention links (a-link and β-link) provide further improvements over the full-link approach, especially in low-data scenarios.

Limitations and Future Work

  • Hyperparameter Tuning: The paper acknowledges that the hyperparameter $\lambda$ (controlling the regularization strength) and its decay rate are important and require careful tuning. While they empirically found $\lambda=2000$ to be optimal for ImageNet 10%, this might vary across datasets and tasks.
  • Analysis of Decay Constant: The authors mention that further analysis of the decay constant for $\lambda$ could improve efficiency, suggesting it as a direction for future work.
  • Generalizability of Teacher Models: While the method is agnostic to the teacher model's dataset, the quality of the teacher's intermediate representations can impact performance, as seen with CIFAR-10 trained models.
  • Future Work: The authors suggest further exploring the analysis of attention links and their relationship with CNNs, potentially leading to more efficient network designs. They also hint at future work to analyze the decay constant for $\lambda$.

Summary updated

Abstract

Although transformer networks are recently employed in various vision tasks with outperforming performance, extensive training data and a lengthy training time are required to train a model to disregard an inductive bias. Using trainable links between the channel-wise spatial attention of a pre-trained Convolutional Neural Network (CNN) and the attention head of Vision Transformers (ViT), we present a regularization technique to improve the training efficiency of ViT. The trainable links are referred to as the attention augmentation module, which is trained simultaneously with ViT, boosting the training of ViT and allowing it to avoid the overfitting issue caused by a lack of data. From the trained attention augmentation module, we can extract the relevant relationship between each CNN activation map and each ViT attention head, and based on this, we also propose an advanced attention augmentation module. Consequently, even with a small amount of data, the suggested method considerably improves the performance of ViT while achieving faster convergence during training.
BibTeX
@inproceedings{jin2022adaptive,
  author = {Heegon Jin and Jongwon Choi},
  title = {{Adaptive Attention Link-based Regularization for Vision Transformers}},
  booktitle = {AVSS},
  year = {2022},
  doi = {10.1109/avss65446.2025.11149958},
}
Figure