Adding Attentiveness to the Neurons in Recurrent Neural Networks

Pengfei Zhang, Jianru Xue, Cuiling Lan, Wenjun Zeng, Zhanning Gao, Nanning Zheng

Abstract

Standard Recurrent Neural Networks (RNNs) treat input vectors as monolithic units, often failing to capture the varying importance of individual elements within those vectors. This paper proposes the Element-wise Attention Gate (EleAttG), a simple yet effective module that adaptively modulates input elements before they enter the core RNN logic. By using a Sigmoid-based attention mechanism, EleAttG allows neurons to independently focus on informative features, leading to significant performance gains in action recognition tasks.

Introduction

Standard Recurrent Neural Networks, including advanced variants like LSTMs and GRUs, are excellent at modeling temporal dynamics. They use internal gates (like the "forget" or "update" gates) to decide how much historical information to keep versus how much new information to let in.

However, these traditional gates typically treat the input vector at a given time step as a single unit or a "whole." They don't account for the fact that some elements or dimensions of that input vector might be more important than others for the task at hand. For example, in skeleton-based action recognition, if a person is "kicking," the coordinates of the feet are far more informative than the coordinates of the head. Existing RNNs lack a built-in mechanism to perform this fine-grained, element-wise prioritization.

The Element-wise Attention Gate (EleAttG)

The authors propose the EleAttG, a fundamental unit that can be wrapped around any RNN block (sRNN, LSTM, or GRU).

The architecture of (a) the general EleAttG-RNN and (b) the specific EleAttG-GRU implementation.

Mechanism

  1. The Attention Vector: At each time step $t$, the gate computes an attention vector $a_t$ based on the current input $x_t$ and the previous hidden state $h_{t-1}$: $$a_t = \text{Sigmoid}(W_{xa}x_t + W_{ha}h_{t-1} + b_a)$$
  2. Element-wise Modulation: This attention vector has the same dimension as the input. The input is then modulated via an element-wise product (Hadamard product): $$\tilde{x}_t = a_t \odot x_t$$
  3. Integration: This "attentive" input $\tilde{x}_t$ is then passed into the standard RNN equations.

Sigmoid vs. Softmax

Unlike traditional attention mechanisms that often use a Softmax function (forcing the weights to sum to 1), this paper uses a Sigmoid function. This allows the importance of one element to be learned independently of others, preventing a "zero-sum game" where increasing focus on one feature automatically decreases focus on another. This design choice was found to be critical; Softmax-based attention actually hurt performance in deeper layers by suppressing useful features unnecessarily.

Experimental Setup

The researchers evaluated their method on four major action recognition datasets:

  • NTU RGB+D: Currently the largest dataset for 3D skeleton-based action recognition (56,880 videos, 60 classes).
  • Northwestern-UCLA (N-UCLA): A smaller 3D skeleton dataset focused on 10 actions.
  • SYSU Human-Object Interaction: A dataset containing 12 actions involving interactions with objects.
  • JHMDB: An RGB-based video dataset used to test the model's generalization to CNN-extracted features.

Key Insights and Results

  • Fine-Grained Granularity: Moving from "input-wise" to "element-wise" control allows the network to filter out noise at the feature level. This is particularly effective for skeleton data where specific joints are key to specific actions.
  • Faster Convergence: Adding the EleAttG acts as a "pre-conditioner" for the learning process. By highlighting important features early, the subsequent RNN layers have an easier time learning the temporal patterns, leading to faster training.
  • State-of-the-Art Performance: On the NTU dataset, adding EleAttGs to a GRU network improved accuracy by 4.6% (Cross-Subject) and 5.6% (Cross-View) over the baseline. The "EleAtt-GRU" model achieved 80.7% accuracy on NTU (Cross-Subject), outperforming many complex, specialized architectures.
  • Efficiency: While it adds parameters, the performance gain is due to the structure of the attention gate. A standard GRU with more neurons still performs worse than a smaller EleAtt-GRU.

Conclusion and Future Work

While the EleAttG provides significant gains, it does introduce some computational overhead in terms of parameters and FLOPs. Additionally, while attention responses in the first layer are interpretable (e.g., highlighting legs for a "kicking" action), modulation in higher, more abstract layers remains difficult to visualize.

Future directions include applying EleAttG to a wider variety of RNN-based tasks beyond action recognition, such as machine translation or speech processing, where element-wise importance may also play a critical role.

Summary updated

Abstract

Recurrent neural networks (RNNs) are capable of modeling the temporal dynamics of complex sequential information. However, the structures of existing RNN neurons mainly focus on controlling the contributions of current and historical information but do not explore the different importance levels of different elements in an input vector of a time slot. We propose adding a simple yet effective Element-wise-Attention Gate (EleAttG) to an RNN block (e.g., all RNN neurons in a network layer) that empowers the RNN neurons to have the attentiveness capability. For an RNN block, an EleAttG is added to adaptively modulate the input by assigning different levels of importance, i.e., attention, to each element/dimension of the input. We refer to an RNN block equipped with an EleAttG as an EleAtt-RNN block. Specifically, the modulation of the input is content adaptive and is performed at fine granularity, being element-wise rather than input-wise. The proposed EleAttG, as an additional fundamental unit, is general and can be applied to any RNN structures, e.g., standard RNN, Long Short-Term Memory (LSTM), or Gated Recurrent Unit (GRU). We demonstrate the effectiveness of the proposed EleAtt-RNN by applying it to the action recognition tasks on both 3D human skeleton data and RGB videos. Experiments show that adding attentiveness through EleAttGs to RNN blocks significantly boosts the power of RNNs.
BibTeX
@inproceedings{zhang2018adding,
  author = {Pengfei Zhang and Jianru Xue and Cuiling Lan and Wenjun Zeng and Zhanning Gao and Nanning Zheng},
  title = {{Adding Attentiveness to the Neurons in Recurrent Neural Networks}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  doi = {10.1007/978-3-030-01240-3\_9},
}
Figure