Strengthening Layer Interaction via Dynamic Layer Attention

Kaishen Wang, Xun Xia, Jian Liu, Zhang Yi, Tao He

Abstract: In recent years, employing layer attention to enhance interaction among hierarchical layers has proven to be a significant advancement in building network structures. In this paper, we delve into the distinction between layer attention and the general attention mechanism, noting that existing layer attention methods achieve layer interaction on fixed feature maps in a static manner. These static layer attention methods limit the ability for context feature extraction among layers. To restore the dynamic context representation capability of the attention mechanism, we propose a Dynamic Layer Attention (DLA) architecture. The DLA comprises dual paths, where the forward path utilizes an improved recurrent neural network block, named Dynamic Sharing Unit (DSU), for context feature extraction. The backward path updates features using these shared context representations. Finally, the attention mechanism is applied to these dynamically refreshed feature maps among layers. Experimental results demonstrate the effectiveness of the proposed DLA architecture, outperforming other state-of-the-art methods in image recognition and object detection tasks. Additionally, the DSU block has been evaluated as an efficient plugin in the proposed DLA architecture.

Introduction

In Deep Convolutional Neural Networks (DCNNs), hierarchical layers capture different levels of abstraction, from low-level textures to high-level semantic objects. To improve performance, researchers often use "layer attention" to help these layers communicate. However, a major flaw exists in current methods: they are fundamentally static.

In a standard network, once an early layer (e.g., Layer 2) produces its feature map, that map remains fixed as the data moves toward deeper layers. When a later layer (e.g., Layer 50) tries to "attend" to earlier ones, it is looking at "stale" information that hasn't benefited from the high-level context discovered by the deeper parts of the network. This limits the efficiency of information flow and often results in the attention mechanism "collapsing" to focus on only one specific layer rather than integrating information across the hierarchy.

The Dynamic Layer Attention (DLA) Framework

The authors propose the Dynamic Layer Attention (DLA) architecture to restore dynamic context representation. This architecture operates via two distinct paths that refresh the network's internal state.

The Dynamic Layer Attention (DLA) architecture overview, illustrating the dual-path structure (Forward and Backward) and the integrated Layer Attention Module.

Dual-Path Architecture

  1. The Forward Path (Context Extraction): As the network processes an image, a Global Average Pooling (GAP) layer extracts a summary vector from each layer. These summaries are fed into a Recurrent Neural Network (RNN) block that accumulates "global context" as it moves deeper into the network hierarchy.
  2. The Backward Path (Feature Refreshing): Once the global context for the current depth is calculated, it is sent back to "refresh" the original feature maps of all preceding layers. This ensures that even early-layer features are updated with a sense of the "big picture" discovered by deeper layers.
  3. The Attention Mechanism: Finally, a standard multi-head attention mechanism is applied to these dynamically refreshed feature maps, allowing for more meaningful interaction between layers.

The Dynamic Sharing Unit (DSU)

To make this process computationally efficient, the authors designed the Dynamic Sharing Unit (DSU). It is a simplified version of an LSTM or DIA unit that discards unnecessary gates (like the output gate) to reduce parameter counts. By using Sigmoid-based normalization, it improves the fusion of new and old information while maintaining a low parameter footprint of just $5C^2/r$ (where $C$ is channels and $r$ is a reduction ratio).

Experimental Results

The DLA architecture was evaluated on several industry-standard benchmarks, consistently outperforming baselines and previous state-of-the-art attention methods like SE-Net and ECA-Net.

  • ImageNet-1K: DLA-L-50 (using a ResNet-50 backbone) achieved a 78.0% top-1 accuracy, a 1.9% improvement over the standard ResNet-50, while using fewer parameters than competing methods.
  • CIFAR-100: DLA-L-56 outperformed the much deeper ResNet-110 by 2.46%, demonstrating superior efficiency.
  • Object Detection (COCO): When used as a backbone for Faster R-CNN, DLA-L-50 improved the Average Precision (AP) by 4.2% over a standard ResNet-50, proving its effectiveness in complex downstream tasks.

Analysis: Breaking the Attention Bottleneck

A key insight of this work is the visualization of attention scores. Static methods often suffer from "attention collapse," where the model focuses nearly 100% of its attention on a single previous layer, effectively ignoring the rest of the hierarchy.

Comparison of attention maps: Static methods (left) often collapse to a single layer, while DLA (right) maintains a distributed, multi-layer focus across the hierarchy.

As shown in the heatmaps above, DLA results in a much more distributed and "healthy" attention map. This confirms that the refreshed features are more useful to the network, allowing it to re-interpret low-level textures in light of high-level semantic objects.

Conclusion

Dynamic Layer Attention moves beyond the limitations of static feature maps by allowing deeper layers to "talk back" to shallower ones before the attention step. This dual-path approach, powered by the efficient DSU block, significantly enhances the representational power of DCNNs with minimal overhead. Future work may explore the application of DLA to more modern architectures like Vision Transformers (ViTs) or its utility in temporal data processing.

Summary updated

Abstract

In recent years, employing layer attention to enhance interaction among hierarchical layers has proven to be a significant advancement in building network structures. In this paper, we delve into the distinction between layer attention and the general attention mechanism, noting that existing layer attention methods achieve layer interaction on fixed feature maps in a static manner. These static layer attention methods limit the ability for context feature extraction among layers. To restore the dynamic context representation capability of the attention mechanism, we propose a Dynamic Layer Attention (DLA) architecture. The DLA comprises dual paths, where the forward path utilizes an improved recurrent neural network block, named Dynamic Sharing Unit (DSU), for context feature extraction. The backward path updates features using these shared context representations. Finally, the attention mechanism is applied to these dynamically refreshed feature maps among layers. Experimental results demonstrate the effectiveness of the proposed DLA architecture, outperforming other state-of-the-art methods in image recognition and object detection tasks. Additionally, the DSU block has been evaluated as an efficient plugin in the proposed DLA architecture. The code is available at https://github.com/tunantu/Dynamic-Layer-attention.
BibTeX
@inproceedings{wang2024strengthening,
  author = {Kaishen Wang and Xun Xia and Jian Liu and Zhang Yi and Tao He},
  title = {{Strengthening Layer Interaction via Dynamic Layer Attention}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=tU0ToHkgLO},
}
Figure