Abstract
Standard Recurrent Neural Networks (RNNs) treat input vectors as monolithic units, often failing to capture the varying importance of individual elements within those vectors. This paper proposes the Element-wise Attention Gate (EleAttG), a simple yet effective module that adaptively modulates input elements before they enter the core RNN logic. By using a Sigmoid-based attention mechanism, EleAttG allows neurons to independently focus on informative features, leading to significant performance gains in action recognition tasks.
Introduction
Standard Recurrent Neural Networks, including advanced variants like LSTMs and GRUs, are excellent at modeling temporal dynamics. They use internal gates (like the "forget" or "update" gates) to decide how much historical information to keep versus how much new information to let in.
However, these traditional gates typically treat the input vector at a given time step as a single unit or a "whole." They don't account for the fact that some elements or dimensions of that input vector might be more important than others for the task at hand. For example, in skeleton-based action recognition, if a person is "kicking," the coordinates of the feet are far more informative than the coordinates of the head. Existing RNNs lack a built-in mechanism to perform this fine-grained, element-wise prioritization.
The Element-wise Attention Gate (EleAttG)
The authors propose the EleAttG, a fundamental unit that can be wrapped around any RNN block (sRNN, LSTM, or GRU).
Mechanism
- The Attention Vector: At each time step $t$, the gate computes an attention vector $a_t$ based on the current input $x_t$ and the previous hidden state $h_{t-1}$: $$a_t = \text{Sigmoid}(W_{xa}x_t + W_{ha}h_{t-1} + b_a)$$
- Element-wise Modulation: This attention vector has the same dimension as the input. The input is then modulated via an element-wise product (Hadamard product): $$\tilde{x}_t = a_t \odot x_t$$
- Integration: This "attentive" input $\tilde{x}_t$ is then passed into the standard RNN equations.
Sigmoid vs. Softmax
Unlike traditional attention mechanisms that often use a Softmax function (forcing the weights to sum to 1), this paper uses a Sigmoid function. This allows the importance of one element to be learned independently of others, preventing a "zero-sum game" where increasing focus on one feature automatically decreases focus on another. This design choice was found to be critical; Softmax-based attention actually hurt performance in deeper layers by suppressing useful features unnecessarily.
Experimental Setup
The researchers evaluated their method on four major action recognition datasets:
- NTU RGB+D: Currently the largest dataset for 3D skeleton-based action recognition (56,880 videos, 60 classes).
- Northwestern-UCLA (N-UCLA): A smaller 3D skeleton dataset focused on 10 actions.
- SYSU Human-Object Interaction: A dataset containing 12 actions involving interactions with objects.
- JHMDB: An RGB-based video dataset used to test the model's generalization to CNN-extracted features.
Key Insights and Results
- Fine-Grained Granularity: Moving from "input-wise" to "element-wise" control allows the network to filter out noise at the feature level. This is particularly effective for skeleton data where specific joints are key to specific actions.
- Faster Convergence: Adding the EleAttG acts as a "pre-conditioner" for the learning process. By highlighting important features early, the subsequent RNN layers have an easier time learning the temporal patterns, leading to faster training.
- State-of-the-Art Performance: On the NTU dataset, adding EleAttGs to a GRU network improved accuracy by 4.6% (Cross-Subject) and 5.6% (Cross-View) over the baseline. The "EleAtt-GRU" model achieved 80.7% accuracy on NTU (Cross-Subject), outperforming many complex, specialized architectures.
- Efficiency: While it adds parameters, the performance gain is due to the structure of the attention gate. A standard GRU with more neurons still performs worse than a smaller EleAtt-GRU.
Conclusion and Future Work
While the EleAttG provides significant gains, it does introduce some computational overhead in terms of parameters and FLOPs. Additionally, while attention responses in the first layer are interpretable (e.g., highlighting legs for a "kicking" action), modulation in higher, more abstract layers remains difficult to visualize.
Future directions include applying EleAttG to a wider variety of RNN-based tasks beyond action recognition, such as machine translation or speech processing, where element-wise importance may also play a critical role.
Summary updated