EAN: An Efficient Attention Module Guided by Normalization for Deep Neural Networks

Jiafeng Li, Zelin Li, Ying WenAAAI 2024 · DOI

Abstract

EAN (Efficient Attention guided by Normalization) is a lightweight, plug-and-play module that unifies feature normalization and attention mechanisms. By repurposing the internal scaling parameters of normalization layers as "importance indicators," EAN achieves state-of-the-art accuracy on vision tasks with virtually zero additional computational overhead or parameter count. This approach bridges the gap between stabilizing training and enhancing feature representation, offering a highly efficient alternative to traditional attention modules.

Introduction

In modern deep learning, two techniques are ubiquitous for boosting performance: Feature Normalization (e.g., Batchnorm or GroupNorm) and Attention Mechanisms (e.g., SE-Net or CBAM). Traditionally, these are treated as separate entities: normalization stabilizes training and speeds up convergence, while attention helps the model focus on salient features.

However, current attention modules often rely on "heavy" components like fully-connected (FC) layers or 1D convolutions to calculate importance weights, adding significant complexity. The authors of EAN propose a novel integration: leveraging the internal statistics of normalization to guide attention. They argue that the scaling parameters already present in normalization layers contain the information needed to perform attention, making external, parameter-heavy modules redundant.

The EAN Architecture

The core innovation of EAN is the Attention Generation (AG) unit, which integrates seamlessly with Group Normalization (GN). The process follows three main stages:

  1. Grouping: The input feature map $\mathcal{X}$ is split into $G$ groups along the channel dimension. This allows the model to capture diverse semantic features in parallel.
  2. The AG Unit: Instead of using a separate neural network to estimate channel importance, the AG unit utilizes the affine transformation parameters ($\gamma$) produced during normalization. A larger $\gamma$ value indicates a channel with higher spatial variance, suggesting it contains more "signal" than noise. These values are refined through $L_2$ normalization and a lightweight adaptation layer to produce a final attention mask.
  3. Aggregation: The re-calibrated feature groups are aggregated to form the final output $\hat{\mathcal{X}}$.
Detailed architecture of the AG unit and its integration with Group Normalization.

By reusing existing parameters, EAN-ResNet50 maintains the exact same parameter count (23.71M) and FLOPs (1.30G) as a standard ResNet50, while significantly improving performance.

Experimental Results

The authors validated EAN across standard benchmarks including ImageNet-1K, CIFAR, and MS COCO, using various backbones like ResNet, MobileNetV2, and EfficientNet.

  • ImageNet-1K: EAN-ResNet50 achieved 77.87% Top-1 accuracy, outperforming both the baseline ResNet50 (76.45%) and the popular SE-ResNet50 (77.31%).
  • Object Detection: On MS COCO, adding EAN to a ResNet50 backbone boosted mAP from 36.4 to 38.4, a substantial gain for a "zero-cost" addition.
  • Qualitative Analysis: Grad-CAM visualizations demonstrate that EAN helps the model focus more tightly on target objects (e.g., a ballplayer or a parachute) while effectively suppressing background noise compared to baseline and SE-Net models.
Grad-CAM visualizations comparing EAN with baseline and other attention modules across different objects and network layers.

Discussion and Conclusion

EAN demonstrates that normalization-guided attention is not only feasible but superior in terms of efficiency. By treating normalization scaling factors as intrinsic importance indicators, EAN achieves faster convergence and higher accuracy without the overhead of external MLPs.

While highly effective for CNNs, future work could explore how this normalization-guided logic translates to Transformer-based architectures, which utilize LayerNorm differently. Currently, EAN stands as a powerful, zero-cost enhancement for a wide range of computer vision tasks, particularly in resource-constrained environments.

Summary updated

BibTeX
@inproceedings{li2024ean,
  author = {Jiafeng Li and Zelin Li and Ying Wen},
  title = {{EAN: An Efficient Attention Module Guided by Normalization for Deep Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28093},
}
Figure