Delta: Deep Learning Transfer using Feature Map with Attention for Convolutional Networks

Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, Jun Huan

Abstract

Transfer learning is a cornerstone of modern deep learning, allowing models pre-trained on large datasets to adapt to smaller, specialized tasks. However, standard fine-tuning often leads to overfitting or the loss of generalizable knowledge. This paper introduces DELTA (DEep Learning Transfer using Feature Map with Attention), a regularization framework that aligns the feature maps of a target network with those of a source network. By incorporating a supervised attention mechanism to prioritize discriminative features, DELTA effectively preserves useful knowledge while allowing the model to adapt to new domains.

Introduction

The primary challenge in transfer learning is balancing the preservation of pre-trained knowledge with the flexibility required for a new task. Traditional methods like $L^2$-SP regularize the model's weights, forcing them to remain close to their initial values. DELTA's authors argue that this "syntactic" approach is overly restrictive because neural networks are highly redundant—different weight configurations can produce identical feature representations. Instead, DELTA proposes "behavioral" regularization, focusing on the feature maps produced by convolutional layers.

Methodology: Behavioral Regularization with Attention

DELTA's architecture is designed to ensure the target model "sees" the world similarly to the successful source model. It consists of two primary innovations:

  1. Feature Map Alignment: Instead of minimizing weight distances, DELTA minimizes the distance between the feature maps of the source and target networks. This ensures that the semantic representations learned during pre-training are maintained.
  2. Supervised Attention Mechanism: Not all source features are relevant to every new task. DELTA uses a "backward variable selection" strategy to assign importance weights to different feature maps. By measuring the performance drop when a specific filter is removed, DELTA identifies which features are critical. High-impact filters receive strong regularization, while low-impact filters are freed for "unactivated channel re-usage" to better fit the new data.
The DELTA framework aligns the feature maps of a target model with a pre-trained source model using a supervised attention mechanism.

Experimental Evaluation

The authors evaluated DELTA across a wide range of benchmarks, including general object recognition (Caltech 256), fine-grained recognition (Stanford Dogs, CUB-200-2011, Food-101), and scene classification (MIT Indoors 67). Experiments were conducted using ResNet-101 and Inception-V3 backbones.

Key Results

DELTA consistently outperformed standard $L^2$ and $L^2$-SP regularization across nearly all tested scenarios. For instance, on the MIT Indoors dataset using Inception-V3, DELTA achieved 78.1% accuracy, a significant improvement over $L^2$-SP's 74.6%. On Stanford Dogs with ResNet-101, it reached a state-of-the-art 88.7%.

Performance comparison of DELTA against standard regularization methods across various image classification benchmarks.

The results demonstrate that while feature map regularization alone (DELTA w/o ATT) is often superior to weight regularization, the addition of the attention mechanism provides the necessary boost to achieve peak performance.

Discussion and Conclusion

DELTA represents a shift from syntactic weight-based regularization to semantic behavior-based regularization. Beyond accuracy gains, DELTA offers smoother convergence and more robust performance across different learning rate schedules. Visualizations further reveal that DELTA focuses more precisely on discriminative object parts compared to noisier activation maps from competing methods.

Future work may involve extending this behavioral alignment to Transformer-based architectures and more complex vision tasks such as object detection and semantic segmentation.

Summary updated

Abstract

Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task. To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied. In this paper, we propose a novel regularized transfer learning framework DELTA, namely DEep Learning Transfer using Feature Map with Attention. Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network. Specifically, in addition to minimizing the empirical loss, DELTA intends to align the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in an supervised learning manner. We evaluate DELTA with the state-of-the-art algorithms, including L2 and L2-SP. The experiment results show that our proposed method outperforms these baselines with higher accuracy for new tasks.
BibTeX
@inproceedings{li2019delta,
  author = {Xingjian Li and Haoyi Xiong and Hanchao Wang and Yuxuan Rao and Liping Liu and Jun Huan},
  title = {{Delta: Deep Learning Transfer using Feature Map with Attention for Convolutional Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=gQJ3nnO398},
}
Figure