Abstract
Transfer learning is a cornerstone of modern deep learning, allowing models pre-trained on large datasets to adapt to smaller, specialized tasks. However, standard fine-tuning often leads to overfitting or the loss of generalizable knowledge. This paper introduces DELTA (DEep Learning Transfer using Feature Map with Attention), a regularization framework that aligns the feature maps of a target network with those of a source network. By incorporating a supervised attention mechanism to prioritize discriminative features, DELTA effectively preserves useful knowledge while allowing the model to adapt to new domains.
Introduction
The primary challenge in transfer learning is balancing the preservation of pre-trained knowledge with the flexibility required for a new task. Traditional methods like $L^2$-SP regularize the model's weights, forcing them to remain close to their initial values. DELTA's authors argue that this "syntactic" approach is overly restrictive because neural networks are highly redundant—different weight configurations can produce identical feature representations. Instead, DELTA proposes "behavioral" regularization, focusing on the feature maps produced by convolutional layers.
Methodology: Behavioral Regularization with Attention
DELTA's architecture is designed to ensure the target model "sees" the world similarly to the successful source model. It consists of two primary innovations:
- Feature Map Alignment: Instead of minimizing weight distances, DELTA minimizes the distance between the feature maps of the source and target networks. This ensures that the semantic representations learned during pre-training are maintained.
- Supervised Attention Mechanism: Not all source features are relevant to every new task. DELTA uses a "backward variable selection" strategy to assign importance weights to different feature maps. By measuring the performance drop when a specific filter is removed, DELTA identifies which features are critical. High-impact filters receive strong regularization, while low-impact filters are freed for "unactivated channel re-usage" to better fit the new data.
Experimental Evaluation
The authors evaluated DELTA across a wide range of benchmarks, including general object recognition (Caltech 256), fine-grained recognition (Stanford Dogs, CUB-200-2011, Food-101), and scene classification (MIT Indoors 67). Experiments were conducted using ResNet-101 and Inception-V3 backbones.
Key Results
DELTA consistently outperformed standard $L^2$ and $L^2$-SP regularization across nearly all tested scenarios. For instance, on the MIT Indoors dataset using Inception-V3, DELTA achieved 78.1% accuracy, a significant improvement over $L^2$-SP's 74.6%. On Stanford Dogs with ResNet-101, it reached a state-of-the-art 88.7%.
The results demonstrate that while feature map regularization alone (DELTA w/o ATT) is often superior to weight regularization, the addition of the attention mechanism provides the necessary boost to achieve peak performance.
Discussion and Conclusion
DELTA represents a shift from syntactic weight-based regularization to semantic behavior-based regularization. Beyond accuracy gains, DELTA offers smoother convergence and more robust performance across different learning rate schedules. Visualizations further reveal that DELTA focuses more precisely on discriminative object parts compared to noisier activation maps from competing methods.
Future work may involve extending this behavioral alignment to Transformer-based architectures and more complex vision tasks such as object detection and semantic segmentation.
Summary updated