Weight Excitation: Built-in Attention Mechanisms in Convolutional Neural Networks

Niamul Quader, Md Mafijul Islam Bhuiyan, Juwei Lu, Peng Dai, Wei Li

Introduction

Modern attention mechanisms like Squeeze-and-Excitation (SE) or CBAM have significantly improved Convolutional Neural Network (CNN) performance by operating on activation maps. However, these methods typically introduce structural changes and increase computational overhead (FLOPs and latency) during inference. This paper introduces Weight Excitation (WE), a training-time attention mechanism that prioritizes important weights without any additional cost at inference. By reparameterizing weights based on their location and magnitude during training, the authors achieve substantial accuracy gains (e.g., +1.3% on ResNet50) while maintaining the original model's inference speed and structure.

Methodology: Weight Excitation Mechanisms

The core idea of Weight Excitation is to apply attention directly to the network's weights during the optimization process. The authors propose two complementary reparameterization techniques:

1. Location-based Weight Excitation (LWE)

LWE identifies important connections based on their structural position (layer, input channel, and output channel). It employs a lightweight subnetwork that takes the weights of a layer as input, performs average pooling, and passes the result through two fully connected layers to generate a "multiplier map." This map scales the weights during the forward pass of training, effectively modulating the gradients during backpropagation and forcing the optimizer to focus on specific locations.

2. Magnitude-based Weight Excitation (MWE)

MWE leverages the observation that weights with higher absolute values often contribute more to network performance. It uses a differentiable log-based activation function, $f_A(\omega)$, to provide relatively larger gains to high-magnitude weights. This allows for fine-grained, weight-wise attention during training.

Architectural diagrams for Location-based (LWE) and Magnitude-based (MWE) weight excitation.

Crucially, both LWE and MWE are only active during training. For deployment, the "excited" weights are used as standard static weights, resulting in a "free lunch" in terms of inference performance.

Key Insights and Empirical Evidence

The Decimation Proof

To validate the hypothesis that weights have varying levels of importance, the authors conducted a "decimation" experiment. By systematically zeroing out weights based on their magnitude percentile, they observed a non-linear relationship between weight removal and performance drop.

Performance degradation as a function of decimated weight percentile, demonstrating the varying importance of weights.

As shown in the figure, removing weights in the highest percentiles (largest magnitudes) causes a catastrophic drop in performance compared to removing lower-magnitude weights. This confirms that magnitude is a reliable proxy for weight importance, justifying the MWE approach.

Structural Importance and Layer Sensitivity

The study found that weight importance varies significantly more in the earlier layers of a CNN than in deeper layers. LWE exploits this by providing more diverse excitation in these critical early stages. Furthermore, the authors discovered that WE is complementary to activation-based attention. Combining LWE and SE yielded superior results, suggesting that weight-level and feature-level attention capture distinct, useful priors.

Experimental Results

The effectiveness of Weight Excitation was validated across multiple datasets (ImageNet, CIFAR100, PASCAL VOC, Cityscapes) and architectures (ResNet, MobileNetV2, VGG19).

  • ImageNet Classification: LWE improved ResNet50 Top-1 accuracy from 75.8% to 77.1% (+1.3%).
  • MobileNetV2: MWE provided a +0.8% boost, demonstrating its utility for lightweight models.
  • Semantic Segmentation: On PASCAL VOC, LWE with a DeepLabv3 backbone improved mean Intersection over Union (mIOU) by 1.3%.
  • Efficiency: While training time increases slightly (~7%), inference speed remains identical to the baseline model.

Conclusion and Limitations

Weight Excitation offers a powerful way to boost CNN performance with zero inference overhead. However, some limitations exist. LWE struggles with depthwise convolutions, where excitation can be "absorbed" by Batch Normalization, though MWE remains effective. Additionally, LWE was found to be detrimental to shift-based 3D convolutions (like TSM) because suppressing input channels can ignore vital temporal signals. Future research may explore whether "unimportant" channels identified by LWE are actually adversarial to the network's performance.

Summary updated

Abstract

We propose novel approaches for simultaneously identifying important weights of a convolutional neural network (ConvNet) and providing more attention to the important weights during training. More formally, we identify two characteristics of a weight, its magnitude and its location, which can be linked with the importance of the weight. By targeting these characteristics of a weight during training, we develop two separate weight excitation (WE) mechanisms via weight reparameterization-based backpropagation modifications. We demonstrate significant improvements over popular baseline ConvNets on multiple computer vision applications using WE (e.g. 1.3% accuracy improvement over ResNet50 baseline on ImageNet image classification, etc.). These improvements come at no extra computational cost or ConvNet structural change during inference. Additionally, including WE methods in a convolution block is straightforward, requiring few lines of extra code. Lastly, WE mechanisms can provide complementary benefits when used with external attention mechanisms such as the popular Squeeze-and-Excitation attention block.
BibTeX
@inproceedings{quader2020weight,
  author = {Niamul Quader and Md Mafijul Islam Bhuiyan and Juwei Lu and Peng Dai and Wei Li},
  title = {{Weight Excitation: Built-in Attention Mechanisms in Convolutional Neural Networks}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58577-8\_6},
}
Figure