TL;DR
EleAtt-RNN introduces a simple "Element-wise-Attention Gate" (EleAttG) that allows Recurrent Neural Networks to adaptively weight individual dimensions of an input vector. By modulating inputs at a fine-grained, element-wise level rather than treating the entire vector as a single unit, this approach significantly boosts performance across action recognition, gesture recognition, and sequential classification tasks with minimal computational overhead.
What Problem Does This Paper Address?
Standard Recurrent Neural Networks (RNNs) and their gated variants like LSTMs and GRUs are excellent at modeling temporal dependencies—deciding what to remember from the past and what to focus on in the present. However, they have a "blind spot" regarding the internal structure of the input vector itself.
In most RNN architectures, gates produce a single scalar value (between 0 and 1) that controls the flow of the entire input vector. This assumes that every element in that vector is equally important at any given time step. In reality, for tasks like skeleton-based action recognition, certain joints (elements of the vector) are far more informative than others for specific actions (e.g., the hands are more important for "making a phone call" than the feet). The authors aim to bridge this gap by adding "attentiveness" to the individual neurons within an RNN block.
Methods
The core innovation is the Element-wise-Attention Gate (EleAttG).
- The Mechanism: For an input vector $x_t$ at time $t$, the EleAttG computes an attention vector $a_t$ of the same dimension. This vector is calculated using the current input and the previous hidden state: $$a_t = \sigma(W_{xa}x_t + W_{ha}h_{t-1} + b_a)$$
- Modulation: The original input is then updated via an element-wise product: $\tilde{x}_t = a_t \odot x_t$.
- Integration: This modulated input $\tilde{x}_t$ is then fed into the standard RNN equations (whether it's a simple RNN, LSTM, or GRU).
The authors emphasize that this is a "shared" design: all neurons in an RNN layer share the same EleAttG to keep the number of parameters manageable. They also highlight that this can be implemented with essentially one additional line of code in most deep learning frameworks.
Datasets
The authors evaluated the EleAtt-RNN on a diverse set of benchmarks:
- 3D Skeleton-based Action Recognition: NTU RGB+D (the largest of its kind), Northwestern-UCLA, and SYSU Human-Object Interaction datasets.
- RGB-based Action Recognition: JHMDB and the RGB portion of NTU RGB+D (using CNN features as inputs).
- Gesture Recognition: Dynamic Hand Gesture 14/28 (DHG) dataset.
- Sequential Classification: The sequential MNIST benchmark, where pixels are fed to the RNN one by one.
Key Insights and Ideas
- Fine-Grained vs. Coarse Control: Traditional RNN gates are "input-wise" (one scalar for the whole vector). EleAttG is "element-wise," allowing the model to suppress noise in specific dimensions while amplifying signal in others.
- Sigmoid > Softmax: Unlike many attention mechanisms that use Softmax to force weights to sum to 1, the authors use a Sigmoid function. This "relaxes" the constraint, allowing multiple elements to be highly important simultaneously and maintaining feature continuity across time steps.
- Universality: The EleAttG is a "fundamental unit." It isn't a new type of RNN but an enhancement that can be "plugged into" any existing recurrent structure (sRNN, LSTM, GRU).
- Efficiency: Adding an EleAttG is more effective than simply increasing the number of neurons or layers in a standard RNN. It provides a better performance-to-parameter ratio.
Key Results
- NTU RGB+D: On the Cross-Subject (CS) protocol, EleAtt-GRU achieved 79.8% accuracy, a significant jump from the baseline GRU's 75.2%.
- Gesture Recognition: On the DHG dataset, the model outperformed previous state-of-the-art methods by 5.2% to 6.3%.
- Sequential MNIST: The model reached 99.2% accuracy, surpassing several complex RNN variants designed specifically for long-range dependencies.
- Interpretability: Visualizations of the attention weights showed the model logically focusing on relevant body parts—for a "kicking" action, the attention weights for leg joints were significantly higher than for arm joints.
Limitations and Future Work
- Shared Attention: While sharing one EleAttG across all neurons in a layer is efficient, the authors acknowledge that, theoretically, each individual neuron could have its own unique attention gate. However, the parameter cost for this is currently prohibitive for high-dimensional inputs.
- Static vs. Dynamic: While the attention is content-adaptive (dynamic), the paper primarily focuses on modulating the input $x_t$. The authors briefly explored modulating the hidden state $h_{t-1}$ but found it less effective, suggesting more research is needed into how attention interacts with internal memory states.
- Future Directions: The authors suggest that EleAttG could be a standard building block for any task involving sequential data where the internal structure of the input vector carries semantic meaning.
Summary updated