Introduction
Modern attention mechanisms like Squeeze-and-Excitation (SE) or CBAM have significantly improved Convolutional Neural Network (CNN) performance by operating on activation maps. However, these methods typically introduce structural changes and increase computational overhead (FLOPs and latency) during inference. This paper introduces Weight Excitation (WE), a training-time attention mechanism that prioritizes important weights without any additional cost at inference. By reparameterizing weights based on their location and magnitude during training, the authors achieve substantial accuracy gains (e.g., +1.3% on ResNet50) while maintaining the original model's inference speed and structure.
Methodology: Weight Excitation Mechanisms
The core idea of Weight Excitation is to apply attention directly to the network's weights during the optimization process. The authors propose two complementary reparameterization techniques:
1. Location-based Weight Excitation (LWE)
LWE identifies important connections based on their structural position (layer, input channel, and output channel). It employs a lightweight subnetwork that takes the weights of a layer as input, performs average pooling, and passes the result through two fully connected layers to generate a "multiplier map." This map scales the weights during the forward pass of training, effectively modulating the gradients during backpropagation and forcing the optimizer to focus on specific locations.
2. Magnitude-based Weight Excitation (MWE)
MWE leverages the observation that weights with higher absolute values often contribute more to network performance. It uses a differentiable log-based activation function, $f_A(\omega)$, to provide relatively larger gains to high-magnitude weights. This allows for fine-grained, weight-wise attention during training.
Crucially, both LWE and MWE are only active during training. For deployment, the "excited" weights are used as standard static weights, resulting in a "free lunch" in terms of inference performance.
Key Insights and Empirical Evidence
The Decimation Proof
To validate the hypothesis that weights have varying levels of importance, the authors conducted a "decimation" experiment. By systematically zeroing out weights based on their magnitude percentile, they observed a non-linear relationship between weight removal and performance drop.
As shown in the figure, removing weights in the highest percentiles (largest magnitudes) causes a catastrophic drop in performance compared to removing lower-magnitude weights. This confirms that magnitude is a reliable proxy for weight importance, justifying the MWE approach.
Structural Importance and Layer Sensitivity
The study found that weight importance varies significantly more in the earlier layers of a CNN than in deeper layers. LWE exploits this by providing more diverse excitation in these critical early stages. Furthermore, the authors discovered that WE is complementary to activation-based attention. Combining LWE and SE yielded superior results, suggesting that weight-level and feature-level attention capture distinct, useful priors.
Experimental Results
The effectiveness of Weight Excitation was validated across multiple datasets (ImageNet, CIFAR100, PASCAL VOC, Cityscapes) and architectures (ResNet, MobileNetV2, VGG19).
- ImageNet Classification: LWE improved ResNet50 Top-1 accuracy from 75.8% to 77.1% (+1.3%).
- MobileNetV2: MWE provided a +0.8% boost, demonstrating its utility for lightweight models.
- Semantic Segmentation: On PASCAL VOC, LWE with a DeepLabv3 backbone improved mean Intersection over Union (mIOU) by 1.3%.
- Efficiency: While training time increases slightly (~7%), inference speed remains identical to the baseline model.
Conclusion and Limitations
Weight Excitation offers a powerful way to boost CNN performance with zero inference overhead. However, some limitations exist. LWE struggles with depthwise convolutions, where excitation can be "absorbed" by Batch Normalization, though MWE remains effective. Additionally, LWE was found to be detrimental to shift-based 3D convolutions (like TSM) because suppressing input channels can ignore vital temporal signals. Future research may explore whether "unimportant" channels identified by LWE are actually adversarial to the network's performance.
Summary updated