Strengthening Layer Interaction via Dynamic Layer Attention

Kaishen Wang, Xun Xia, Jian Liu, Zhang Yi, Tao He

Abstract

In recent years, employing layer attention to enhance interaction among hierarchical layers has proven to be a significant advancement in building network structures. In this paper, we delve into the distinction between layer attention and the general attention mechanism, noting that existing layer attention methods achieve layer interaction on fixed feature maps in a static manner. These static layer attention methods limit the ability for context feature extraction among layers. To restore the dynamic context representation capability of the attention mechanism, we propose a Dynamic Layer Attention (DLA) architecture. The DLA comprises dual paths, where the forward path utilizes an improved recurrent neural network block, named Dynamic Sharing Unit (DSU), for context feature extraction. The backward path updates features using these shared context representations. Finally, the attention mechanism is applied to these dynamically refreshed feature maps among layers. Experimental results demonstrate the effectiveness of the proposed DLA architecture, outperforming other state-of-the-art methods in image recognition and object detection tasks. Additionally, the DSU block has been evaluated as an efficient plugin in the proposed DLA architecture.

Introduction

Deep Convolutional Neural Networks (DCNNs) rely heavily on effective layer interaction to propagate information across hierarchical levels. While traditional architectures like ResNet and DenseNet utilize skip or dense connections, recent research has shifted toward layer attention. This approach treats the features of each layer as "tokens"—analogous to words in a sequence—allowing the network to learn which layers contribute most to the final representation.

However, a critical limitation persists: existing layer attention mechanisms are static. In a standard feed-forward process, once a layer's output is computed, it remains fixed. When a subsequent attention module queries these "frozen" features, it lacks the ability to capture evolving contextual information. This paper addresses this bottleneck by introducing Dynamic Layer Attention (DLA), a framework that enables continuous feature refreshing across the network's depth.

The Dynamic Layer Attention (DLA) Architecture

The core of the DLA architecture is a dual-path system designed to break the static nature of hierarchical layers.

Dual-Path Mechanism

  1. Forward Path (Context Extraction): A specialized Recurrent Neural Network (RNN) traverses the network depth-wise. It aggregates global information from each layer to construct a "context representation" ($c^l$).
  2. Backward Path (Feature Refreshing): This context is then propagated back to preceding layers. Each layer uses this shared global context to "refresh" its original feature maps through element-wise multiplication, ensuring that early-layer representations are informed by deeper network insights.
The DLA architecture (right) and the internal Layer Attention Module (left).

Dynamic Sharing Unit (DSU)

To maintain computational efficiency, the authors introduced the Dynamic Sharing Unit (DSU). Unlike standard LSTMs, which are parameter-heavy, the DSU is a streamlined RNN block. By removing the output gate and merging memory and hidden vectors, the DSU focuses exclusively on depth-wise context extraction. This design allows DLA to outperform established attention modules like SE or CBAM while often requiring fewer parameters.

Experimental Evaluation

The DLA architecture was rigorously evaluated across multiple benchmarks, demonstrating superior performance and efficiency:

  • ImageNet-1K: The lightweight DLA-L variant improved ResNet-50's top-1 accuracy by 1.9% and ResNet-101's by 1.5%, outperforming established methods like SE-Net.
  • CIFAR-100: DLA-L achieved a 3.41% improvement over the ResNet-110 baseline.
  • Object Detection (COCO): Integrating DLA-L into a Faster R-CNN with a ResNet-50 backbone yielded a significant 4.2% increase in Average Precision (AP).

Analysis of Dynamic Interaction

The primary advantage of DLA is its ability to generate dynamic attention maps. By refreshing features, the attention mechanism can focus on more relevant layer interactions that static methods miss.

Visualization of attention maps: (a) Static Layer Attention vs. (b) Dynamic Layer Attention.

As shown in the comparison above, the dynamic approach produces a more nuanced and distributed attention pattern across layers, whereas static attention tends to be more rigid. This flexibility allows the network to better manage information bottlenecks, especially as depth increases.

Conclusion

Dynamic Layer Attention represents a shift from viewing layers as static tokens to seeing them as evolving representations. By leveraging the DSU for efficient depth-wise context management, DLA significantly enhances the representational power of DCNNs. Future work may explore the application of these dynamic principles to Vision Transformers (ViTs) and further optimize the sequential overhead inherent in dual-path architectures.

Summary updated

Abstract

In recent years, employing layer attention to enhance interaction among hierarchical layers has proven to be a significant advancement in building network structures. In this paper, we delve into the distinction between layer attention and the general attention mechanism, noting that existing layer attention methods achieve layer interaction on fixed feature maps in a static manner. These static layer attention methods limit the ability for context feature extraction among layers. To restore the dynamic context representation capability of the attention mechanism, we propose a Dynamic Layer Attention (DLA) architecture. The DLA comprises dual paths, where the forward path utilizes an improved recurrent neural network block, named Dynamic Sharing Unit (DSU), for context feature extraction. The backward path updates features using these shared context representations. Finally, the attention mechanism is applied to these dynamically refreshed feature maps among layers. Experimental results demonstrate the effectiveness of the proposed DLA architecture, outperforming other state-of-the-art methods in image recognition and object detection tasks. Additionally, the DSU block has been evaluated as an efficient plugin in the proposed DLA architecture.The code is available at https://github.com/tunantu/Dynamic-Layer-Attention.
BibTeX
@inproceedings{wang2024strengthening,
  author = {Kaishen Wang and Xun Xia and Jian Liu and Zhang Yi and Tao He},
  title = {{Strengthening Layer Interaction via Dynamic Layer Attention}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=tU0ToHkgLO},
}
Figure