Adding Attentiveness to the Neurons in Recurrent Neural Networks

Pengfei Zhang, Jianru Xue, Cuiling Lan, Wenjun Zeng, Zhanning Gao, Nanning Zheng

Abstract

Recurrent neural networks (RNNs) are widely used for modeling temporal dynamics in sequential data. However, standard RNN architectures primarily focus on controlling the flow of information across time steps rather than evaluating the relative importance of individual elements within an input vector. We propose the Element-wise Attention Gate (EleAttG), a simple yet effective unit that empowers RNN neurons with attentiveness. By adaptively modulating input elements at a fine granularity, EleAttG allows the network to focus on the most relevant features for a given task. This general-purpose unit can be integrated into any standard RNN, LSTM, or GRU block. Our experiments on action recognition tasks using 3D skeleton data and RGB videos demonstrate that EleAtt-RNNs significantly outperform baseline models in both accuracy and convergence speed.

The Challenge of Coarse Gating

Traditional RNN architectures like Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) utilize gates to manage information flow. While effective, these gates typically apply a single scaling factor to entire vectors or hidden states. In many real-world signals, such as 3D human skeleton coordinates, not every dimension of the input is equally informative at every moment. For instance, during a "kicking" action, the coordinates of the feet are critical, while the coordinates of the head may be less relevant. Standard RNNs lack a built-in mechanism to perform this fine-grained, element-wise filtering, often forcing the hidden state to compensate for noisy or irrelevant input dimensions.

Element-wise Attention Gates (EleAttG)

To address this, we introduce the EleAttG, a fundamental unit that can be "plugged into" any RNN block. The gate calculates an attention vector $a_t$ based on the current input $x_t$ and the previous hidden state $h_{t-1}$:

$$a_t = \sigma(W_{xa}x_t + W_{ha}h_{t-1} + b_a)$$

This attention vector, which shares the same dimensionality as the input, is multiplied element-wise with the raw input to create a refined input $\tilde{x}_t = a_t \odot x_t$. This refined input is then passed to the standard RNN logic.

Figure 1: (a) An EleAtt-RNN block where EleAttG is added to a general RNN. (b) An EleAtt-GRU block where EleAttG is integrated into a GRU cell.

Key Design Choice: Sigmoid vs. Softmax

Unlike many attention mechanisms that use a Softmax function to force weights to sum to one, EleAttG utilizes a Sigmoid function. This allows the network to judge each element's importance independently. If multiple elements are crucial, they can all receive high weights; conversely, if the entire input is noisy, all elements can be suppressed. The authors found that the "sum-to-one" constraint of Softmax often hurts performance by forcing a trade-off between relevant features.

Experimental Evaluation

We evaluated the EleAtt-RNN on several challenging action recognition datasets, including NTU RGB+D (the largest 3D skeleton dataset), Northwestern-UCLA, and JHMDB (RGB video).

Performance and Convergence

  • Significant Accuracy Boost: On the NTU dataset, adding EleAttGs to a GRU network improved accuracy by 4.6% to 5.6% over the baseline. The EleAtt-GRU achieved state-of-the-art results (80.7% CS and 88.4% CV), outperforming more complex, specialized architectures.
  • Faster Convergence: Training curves indicate that EleAtt-RNNs reach lower loss levels significantly faster than standard RNNs, suggesting that the attention mechanism simplifies the optimization landscape.
  • Efficiency: Adding an EleAttG is more effective than simply increasing the width or depth of the network, providing a superior performance-to-parameter ratio.

Conclusion

The Element-wise Attention Gate provides a simple, general, and highly effective way to enhance the representational power of RNNs. By shifting from coarse, input-wise gating to fine-grained, element-wise attention, EleAtt-RNNs can better handle high-dimensional sequential data. Given its ease of implementation—requiring only a single additional line of code in many frameworks—it serves as a powerful upgrade for existing recurrent neural network pipelines.

Summary updated

Abstract

Recurrent neural networks (RNNs) are capable of modeling the temporal dynamics of complex sequential information. However, the structures of existing RNN neurons mainly focus on controlling the contributions of current and historical information but do not explore the different importance levels of different elements in an input vector of a time slot. We propose adding a simple yet effective Element-wiseAttention Gate (EleAttG) to an RNN block (e.g., all RNN neurons in a network layer) that empowers the RNN neurons to have the attentiveness capability. For an RNN block, an EleAttG is added to adaptively modulate the input by assigning different levels of importance, i.e., attention, to each element/dimension of the input. We refer to an RNN block equipped with an EleAttG as an EleAtt-RNN block. Specifically, the modulation of the input is content adaptive and is performed at fine granularity, being element-wise rather than input-wise. The proposed EleAttG, as an additional fundamental unit, is general and can be applied to any RNN structures, e.g., standard RNN, Long Short-Term Memory (LSTM), or Gated Recurrent Unit (GRU). We demonstrate the effectiveness of the proposed EleAtt-RNN by applying it to the action recognition tasks on both 3D human skeleton data and RGB videos. Experiments show that adding attentiveness through EleAttGs to RNN blocks significantly boosts the power of RNNs.
BibTeX
@inproceedings{zhang2018adding,
  author = {Pengfei Zhang and Jianru Xue and Cuiling Lan and Wenjun Zeng and Zhanning Gao and Nanning Zheng},
  title = {{Adding Attentiveness to the Neurons in Recurrent Neural Networks}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  doi = {10.1007/978-3-030-01240-3\_9},
}
Figure