EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networks

Pengfei Zhang, Jianru Xue, Cuiling Lan, Wenjun Zeng, Zhanning Gao, Nanning Zheng

Abstract

Recurrent neural networks (RNNs) are capable of modeling temporal dependencies of complex sequential data. In general, current available structures of RNNs tend to concentrate on controlling the contributions of current and previous information. However, the exploration of different importance levels of different elements within an input vector is always ignored. We propose a simple yet effective Element-wise-Attention Gate (EleAttG), which can be easily added to an RNN block (e.g. all RNN neurons in an RNN layer), to empower the RNN neurons to have attentiveness capability. For an RNN block, an EleAttG is used for adaptively modulating the input by assigning different levels of importance, i.e., attention, to each element/dimension of the input. We refer to an RNN block equipped with an EleAttG as an EleAtt-RNN block. Instead of modulating the input as a whole, the EleAttG modulates the input at fine granularity, i.e., element-wise, and the modulation is content adaptive. The proposed EleAttG, as an additional fundamental unit, is general and can be applied to any RNN structures, e.g., standard RNN, Long Short-Term Memory (LSTM), or Gated Recurrent Unit (GRU). We demonstrate the effectiveness of the proposed EleAtt-RNN by applying it to different tasks including the action recognition, from both skeleton-based data and RGB videos, gesture recognition, and sequential MNIST classification. Experiments show that adding attentiveness through EleAttGs to RNN blocks significantly improves the power of RNNs.

Introduction

Standard Recurrent Neural Networks (RNNs) and their gated variants like LSTMs and GRUs are excellent at modeling temporal dependencies—deciding what to remember from the past and what to focus on in the present. However, they typically treat the input vector as a single unit. In most RNN architectures, gates produce a single scalar value that controls the flow of the entire input vector, assuming every element is equally important at any given time step.

In reality, for tasks like skeleton-based action recognition, certain joints (elements of the vector) are far more informative than others for specific actions. For example, the hands are more important for "making a phone call" than the feet. The proposed EleAtt-RNN addresses this "blind spot" by adding "attentiveness" to the individual neurons within an RNN block, allowing the model to adaptively weight individual dimensions of an input vector.

The EleAtt-RNN Architecture

The core innovation is the Element-wise-Attention Gate (EleAttG). This mechanism allows for fine-grained control over the input features.

For an input vector $x_t$ at time $t$, the EleAttG computes an attention vector $a_t$ of the same dimension. This vector is calculated using the current input and the previous hidden state: $$a_t = \sigma(W_{xa}x_t + W_{ha}h_{t-1} + b_a)$$

The original input is then updated via an element-wise product: $\tilde{x}_t = a_t \odot x_t$. This modulated input $\tilde{x}_t$ is then fed into the standard RNN equations.

Architecture of (a) an EleAtt-RNN block and (b) an EleAtt-GRU block.

Key Design Principles

  • Fine-Grained Control: Unlike traditional "input-wise" gates, EleAttG is "element-wise," allowing the model to suppress noise in specific dimensions while amplifying signal in others.
  • Sigmoid Activation: The authors use a Sigmoid function rather than Softmax. This "relaxes" the constraint, allowing multiple elements to be highly important simultaneously and maintaining feature continuity across time steps.
  • Universality and Efficiency: The EleAttG is a "fundamental unit" that can be plugged into any existing recurrent structure (sRNN, LSTM, GRU). In most frameworks, it can be implemented with just one additional line of code. To keep parameters manageable, all neurons in an RNN layer share the same EleAttG.

Experimental Evaluation

The authors evaluated the EleAtt-RNN on a diverse set of benchmarks including 3D skeleton-based action recognition (NTU RGB+D, Northwestern-UCLA), RGB-based action recognition (JHMDB), gesture recognition (DHG), and sequential MNIST classification.

Performance comparison on NTU-CS, NTU-CV, and N-UCLA datasets.

Key Results

  • Action Recognition: On the NTU RGB+D Cross-Subject (CS) protocol, the EleAtt-GRU achieved 79.8% accuracy, a significant improvement over the baseline GRU's 75.2%. Similar gains were observed across other datasets like NTU-CV and Northwestern-UCLA.
  • Gesture Recognition: On the DHG dataset, the model outperformed previous state-of-the-art methods by 5.2% to 6.3%.
  • Sequential MNIST: The model reached 99.2% accuracy, surpassing several complex RNN variants designed for long-range dependencies.
  • Interpretability: Visualizations confirmed that the model logically focuses on relevant body parts—for a "kicking" action, attention weights for leg joints were significantly higher than for arm joints.

Conclusion

EleAtt-RNN introduces a simple yet powerful way to enhance RNNs with element-wise attentiveness. By modulating inputs at a fine granularity, it significantly boosts performance across various sequential tasks with minimal computational overhead. Future work may explore more complex attention sharing schemes or the modulation of internal hidden states to further refine the model's focus.

Summary updated

Abstract

Recurrent neural networks (RNNs) are capable of modeling temporal dependencies of complex sequential data. In general, current available structures of RNNs tend to concentrate on controlling the contributions of current and previous information. However, the exploration of different importance levels of different elements within an input vector is always ignored. We propose a simple yet effective Element-wise-Attention Gate (EleAttG), which can be easily added to an RNN block (e.g. all RNN neurons in an RNN layer), to empower the RNN neurons to have attentiveness capability. For an RNN block, an EleAttG is used for adaptively modulating the input by assigning different levels of importance, i.e., attention, to each element/dimension of the input. We refer to an RNN block equipped with an EleAttG as an EleAtt-RNN block. Instead of modulating the input as a whole, the EleAttG modulates the input at fine granularity, i.e., element-wise, and the modulation is content adaptive. The proposed EleAttG, as an additional fundamental unit, is general and can be applied to any RNN structures, e.g., standard RNN, Long Short-Term Memory (LSTM), or Gated Recurrent Unit (GRU). We demonstrate the effectiveness of the proposed EleAtt-RNN by applying it to different tasks including the action recognition, from both skeleton-based data and RGB videos, gesture recognition, and sequential MNIST classification. Experiments show that adding attentiveness through EleAttGs to RNN blocks significantly improves the power of RNNs.
BibTeX
@article{zhang2020eleatt,
  author = {Pengfei Zhang and Jianru Xue and Cuiling Lan and Wenjun Zeng and Zhanning Gao and Nanning Zheng},
  title = {{EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networks}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=6LzOXqmx1E},
}
Figure