M2A: Motion Aware Attention for Accurate Video Action Recognition

Brennan Gebotys, Alexander Wong, David A. Clausi

TL;DR

This paper introduces Motion Aware Attention (M2A), a novel mechanism that explicitly incorporates motion information into attention models for video action recognition. M2A significantly boosts performance across various backbone architectures, demonstrating that combining motion and attention is crucial for accurately understanding actions in videos.

What Problem Does This Paper Address?

Video action recognition aims to identify what actions are happening in video sequences. While attention mechanisms have revolutionized many areas of machine learning by allowing models to dynamically focus on important parts of the input, their application in video action recognition has primarily focused on spatial (within a frame) and temporal (across frames) attention. However, these existing methods often overlook a critical feature of videos: motion. The authors argue that current attention mechanisms, by only considering visual features, fail to leverage the inherent motion cues that are vital for distinguishing between actions. This gap limits the accuracy of video action recognition models, especially for tasks that heavily rely on dynamic interactions.

Methods

The core contribution of this paper is the Motion Aware Attention (M2A) mechanism. M2A is designed to be easily integrated into existing deep learning architectures for video action recognition. It operates in two main stages:

  1. Motion Block: This stage explicitly extracts motion information. It does this by computing the difference between consecutive frames after an initial convolution and layer normalization. This process effectively captures the changes in visual features over time, emulating motion.
  2. Attention Block: This stage leverages the extracted motion information. It applies self-attention across the temporal axis to focus on relevant motion patterns found across frames. This is followed by a skip connection with the convolved input and a sigmoid activation to refine features.

M2A is designed to be modular, allowing it to be inserted into various backbone architectures like 2D-ResNet, 2D-MobileNet, and I3D-inflated-ResNet. The authors also explore combining M2A with other state-of-the-art (SOTA) attention and temporal mechanisms.

Datasets

The primary dataset used for evaluation in this paper is the Something-Something V1 (SSv1) dataset. This is a standard benchmark for video action recognition, known for its focus on human-object interactions and temporal reasoning. The paper also mentions using ImageNet for pre-training backbone networks.

Key Insights and Ideas

  • Motion is Crucial for Attention: The most significant insight is that simply applying attention to visual features is insufficient for accurate video action recognition. Explicitly incorporating motion information into the attention mechanism leads to substantial performance gains.
  • Synergy of Motion and Attention: The paper demonstrates that the combination of motion and attention (M2A) is more effective than either component alone. M2A allows the model to focus on where and when motion occurs, leading to better action understanding.
  • Modularity and Efficiency: M2A is designed to be easily pluggable into existing architectures with only a marginal increase in computational complexity, making it a practical solution.
  • Complementary to Other Mechanisms: M2A can be effectively combined with other SOTA temporal mechanisms, further enhancing performance and suggesting its role as a complementary component rather than a replacement.

Key Results

  • Significant Accuracy Improvements: M2A achieved improvements of +15% to +26% in top-1 accuracy across different backbone architectures (2D-ResNet18, 2D-MobileNetV2, I3D-inflated-ResNet18) compared to baseline models without explicit motion-aware attention.
  • Outperforming SOTA: When compared to other SOTA motion-only or attention-only mechanisms on the SSv1 benchmark, M2A achieved improvements of up to +60% in top-1 accuracy for specific classes.
  • Efficiency: The inclusion of M2A resulted in only a small increase in computational cost (e.g., +1.6% GMACs/video for 2D-ResNet18), highlighting its efficiency.
  • Qualitative Improvements: Grad-CAM visualizations showed that M2A enables the model to attend to a larger portion of the video frames, capturing more relevant motion cues, leading to correct action classification where single-component methods failed.
  • Resource-Constrained Viability: M2A demonstrated strong performance on the lightweight 2D-MobileNetV2 backbone, achieving higher accuracy with significantly lower computational cost than 3D CNNs, making it suitable for resource-constrained environments.

Limitations and Future Work

  • 3D CNNs: The paper notes that M2A provides only a small improvement when integrated with 3D CNNs (like I3D-inflated-ResNet18). This is attributed to 3D CNNs already modeling temporal aspects effectively, suggesting M2A is more beneficial for 2D architectures that require explicit temporal modeling.
  • Comparison with Combined SOTA: While M2A was compared to individual SOTA mechanisms, the paper also explored combining M2A with TSM (Temporal Shift Module), showing that M2A + TSM outperforms TSM alone, indicating M2A's complementary nature.
  • Future Work: The authors suggest that their research helps develop more accurate and efficient temporal mechanisms for video action recognition. The findings imply that further exploration into combining motion and attention, potentially with more advanced temporal modeling techniques, could yield even better results. The modularity of M2A also opens avenues for its application in other video understanding tasks.

Summary updated

Abstract

Advancements in attention mechanisms have led to significant performance improvements in a variety of areas in machine learning due to its ability to enable the dynamic modeling of temporal sequences. A particular area in computer vision that is likely to benefit greatly from the incorporation of attention mechanisms in video action recognition. However, much of the current research's focus on attention mechanisms have been on spatial and temporal attention, which are unable to take advantage of the inherent motion found in videos. Motivated by this, we develop a new attention mechanism called Motion Aware Attention (M2A) that explicitly incorporates motion characteristics. More specifically, M2A extracts motion information between consecutive frames and utilizes attention to focus on the motion patterns found across frames to accurately recognize actions in videos. The proposed M2A mechanism is simple to implement and can be easily incorporated into any neural network backbone architecture. We show that incorporating motion mechanisms with attention mechanisms using the proposed M2A mechanism can lead to a +15% to +26% improvement in top-1 accuracy across different backbone architectures, with only a small increase in computational complexity. We further compared the performance of M2A with other state-of-the-art motion and attention mechanisms on the Something-Something V1 video action recognition benchmark. Experimental results showed that M2A can lead to further improvements when combined with other temporal mechanisms and that it outperforms other motion-only or attention-only mechanisms by as much as +60% in top-1 accuracy for specific classes in the benchmark.
BibTeX
@inproceedings{gebotys2021m,
  author = {Brennan Gebotys and Alexander Wong and David A. Clausi},
  title = {{M2A: Motion Aware Attention for Accurate Video Action Recognition}},
  booktitle = {CRV},
  year = {2021},
  doi = {10.1109/crv55824.2022.00019},
}
Figure