Shift-and-Balance Attention

Chunjie Luo, Jianfeng Zhan, Tianshu Hao, Lei Wang, Wanling GaoarXiv 2021 · arXiv

TL;DR

This paper introduces "Shift-and-Balance" (SB) attention, a novel mechanism that enhances deep neural networks by adaptively balancing the contributions of a "trunk" feature branch and a lightweight "attention" branch. Unlike previous methods that scale attention, SB adds it, allowing for better control and preventing channel inactivation, leading to improved accuracy and efficiency across various architectures and datasets.

What Problem Does This Paper Address?

Deep neural networks (DNNs) have achieved remarkable success, but their accuracy often comes at the cost of significant computational and memory resources, limiting their deployment on resource-constrained devices. Attention mechanisms offer a way to improve DNN performance with minimal overhead. The Squeeze-and-Excite (SE) attention module, for instance, uses a lightweight branch to model channel-wise relationships. However, SE's approach of scaling the trunk branch by the attention output can be overly sensitive to the balance between the two branches. This can lead to channel inactivation (when the attention output is near zero) or gradient vanishing during training, especially in lightweight networks. More advanced methods like Dynamic Convolution (DyConv) address this by applying attention to convolutional kernels, but they introduce substantial memory and computational overheads and are not always compatible with existing deep learning libraries. This paper aims to develop an attention mechanism that offers the benefits of improved accuracy and efficiency without the drawbacks of SE and the high costs of DyConv.

Methods

The core contribution is the Shift-and-Balance (SB) attention mechanism. Unlike SE, which scales the trunk branch $T(x)$ by the attention output $A(x)$ (i.e., $y = A(x) \cdot T(x)$), SB adds the attention output to the trunk branch, modulated by a learned control factor $\lambda$: $y = T(x) + \lambda A(x)$.

Here's a breakdown of the SB method:

  • Trunk Branch ($T(x)$): This is the standard convolutional path of the network, responsible for feature extraction.
  • Attention Branch ($A(x)$): This is a lightweight network, typically consisting of global average pooling (GAP) followed by a multi-layer perceptron (MLP).
  • Gate Function: The output of the attention branch is passed through a gate function. The paper defaults to using the Tanh function, which provides a symmetric output range. This is a key difference from SE's Sigmoid.
  • Learned Control Factor ($\lambda$): This is a crucial learnable parameter that controls the influence of the attention branch. It allows the network to dynamically balance the contributions of the trunk and attention branches.
  • Integration: The scaled attention output ($\lambda A(x)$) is added to the trunk branch output $T(x)$. This additive approach prevents complete channel inactivation and avoids gradient vanishing issues associated with SE's multiplicative scaling.

The paper also discusses the overheads of SB, which are comparable to SE, primarily involving the auxiliary network for attention and the integration step (addition vs. multiplication).

Datasets

The paper evaluates SB on the following datasets:

  • ImageNet: A large-scale dataset for image classification.
  • PASCAL VOC2007 + 2012: Used for object detection tasks.
  • CIFAR-10: A common dataset for image classification.

Key Insights and Ideas

  • Additive Attention for Better Control: Shifting from multiplicative scaling (SE) to additive integration ($\lambda A(x) + T(x)$) is a fundamental insight. This additive approach provides a more robust way to balance contributions and avoids the extreme outcomes of channel inactivation or saturation.
  • Learned $\lambda$ as a Balancing Act: The introduction of a learned scalar $\lambda$ is key. It allows the network to learn the optimal degree of attention influence for each channel and layer, rather than relying solely on a fixed gating mechanism.
  • Tanh for Symmetry: Using Tanh as the gate function offers a symmetric output range, which can be beneficial for controlling the "shift" applied by the attention branch. However, the paper also notes that other gate functions (like Sigmoid) can be used, and in some cases, omitting the gate entirely can still yield good results if $\lambda$ is well-learned.
  • Layer-wise Application Benefits: Applying SB attention across more layers, especially in lightweight architectures, consistently improves performance. This suggests that SB effectively enhances the representational capacity of networks without introducing prohibitive costs.
  • Regularization is Key for Larger Models: While SB generally improves accuracy, especially in smaller models, for larger models on datasets like ImageNet, regularization (like Dropout) becomes important when applying SB across many layers to prevent overfitting.

Key Results

  • ImageNet: SB consistently outperforms static models and SE across various MobileNetV2 width multipliers. It also achieves better or comparable accuracy to DyConv, especially on smaller networks, with significantly lower computational overhead. For example, on MobileNetV2_x0.35, SB achieves 62.290% accuracy with 60.2M MAdds, while DyConv achieves 62.136% with 62.0M MAdds.
  • PASCAL VOC: SB significantly improves the mean Average Precision (mAP) compared to static SSD-MobileNetV2. Crucially, SE applied to all convolutional layers severely degrades performance, whereas SB consistently enhances it. For instance, on SSD-MobileNetV2_x0.35, SB achieves 46.795% mAP, while SE drops to 11.025%.
  • CIFAR-10: SB demonstrates strong performance, outperforming static models and DyConv. Applying SB in more layers generally leads to higher accuracy.
  • Other Architectures: SB also shows improvements on ShuffleNetV2_x0.5 and MnasNet_x0.5, where SE actually degrades performance.
  • Ablation Studies:
    • Initial values of $\lambda$ around 0.01-0.1 generally yield the best results.
  • Tanh is a robust gate function, but omitting the gate can also work well.
  • Increasing the number of hidden units in the attention branch can improve accuracy up to a point, after which overfitting may occur.

Limitations and Future Work

  • Overfitting in Larger Models: The paper notes that when SB is applied in many layers on large-scale datasets, it can increase the risk of overfitting. The authors address this by using Dropout, suggesting that further research into regularization techniques for SB could be beneficial.
  • Gate Function Choice: While Tanh is a good default, the paper shows that other gate functions can be used, and in some cases, no gate function is optimal. Exploring the theoretical implications of different gate functions and their interaction with $\lambda$ could be a future direction.
  • Computational Cost of Auxiliary Network: Although SB's overall overhead is low, the auxiliary network for attention still contributes to the computational cost. Future work could focus on further optimizing this auxiliary network.
  • Generalization to Other Tasks: While evaluated on classification, object detection, and segmentation, exploring SB's effectiveness on a wider range of computer vision tasks and architectures would be valuable.

Summary updated

BibTeX
@misc{luo2021shift,
  author = {Chunjie Luo and Jianfeng Zhan and Tianshu Hao and Lei Wang and Wanling Gao},
  title = {{Shift-and-Balance Attention}},
  year = {2021},
  eprint = {2103.13080},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.13080},
}
Figure