Action Recognition by an Attention-Aware Temporal Weighted Convolutional Neural Network

Le Wang, Jinliang Zang, Qilin Zhang, Zhenxing Niu, Gang Hua, Nanning Zheng

TL;DR

This paper introduces the Attention-aware Temporal Weighted CNN (ATW CNN), a novel multi-stream neural network that leverages a temporal attention mechanism to improve video action recognition. By adaptively weighting video segments based on their semantic relevance, the ATW CNN achieves state-of-the-art performance on benchmark datasets, demonstrating the power of focused temporal analysis.

What Problem Does This Paper Address?

Action recognition in videos is a fundamental yet challenging task in computer vision. While Convolutional Neural Networks (CNNs) have revolutionized image analysis, effectively incorporating the temporal dimension of video data into CNNs remains an active research area. Existing methods often struggle with the variable length of videos and the redundancy or irrelevance of certain frames, leading to suboptimal performance. The challenge lies in developing methods that can efficiently and effectively capture the crucial temporal dynamics of actions without being overwhelmed by noise or irrelevant information.

Methods

The ATW CNN employs a multi-stream architecture, processing video data through three parallel streams: one for RGB frames, one for optical flow, and one for "warped" optical flow (which accounts for camera motion). Each stream utilizes a pre-trained ResNet-101 as its base CNN to extract spatial features. The core innovation is the temporal attention mechanism. This mechanism, inspired by attention models in natural language processing, assigns a "temporal weight" to each video snippet (a collection of frames from a segment). These weights are learned via a multi-layer perceptron and normalized using a softmax function. Snippets deemed more semantically important receive higher weights, effectively allowing the network to focus on critical moments within an action. The outputs from the three streams are then fused using a weighted average, with the learned temporal weights guiding the final prediction. The entire ATW CNN is designed to be end-to-end trainable via stochastic gradient descent (SGD) with back-propagation.

Datasets

The ATW CNN was evaluated on two widely used benchmark datasets for action recognition:

  • UCF-101: Contains 13,320 YouTube video clips across 101 action categories, including sports and human-object interactions.
  • HMDB-51: A more challenging dataset with 6,766 video clips across 51 action categories, often featuring more complex real-world actions from movies.

The paper does not introduce new datasets.

Key Insights and Ideas

  • Temporal Attention is Crucial: The paper strongly suggests that simply processing all video frames equally is inefficient. By learning to assign differential importance to different temporal segments, the model can significantly boost performance.
  • Multi-Modality Synergy: Combining RGB, optical flow, and warped optical flow provides complementary information that enhances action recognition. Warped optical flow, in particular, helps mitigate the impact of camera motion.
  • End-to-End Learning: The ability to train the entire network, including the attention weights, end-to-end simplifies the training process and allows for optimal parameter tuning.
  • Focus on Discriminative Snippets: The attention mechanism effectively filters out less informative or noisy frames, allowing the model to concentrate on the most semantically relevant parts of an action.

Key Results

The ATW CNN achieved state-of-the-art performance on both the UCF-101 and HMDB-51 datasets. Specifically:

  • On the UCF-101 dataset, the proposed ATW CNN achieved an accuracy of 94.6%.
  • On the HMDB-51 dataset, the proposed ATW CNN achieved an accuracy of 70.5%.

These results surpassed 14 other existing state-of-the-art methods at the time of publication. Visualizations of the attention weights demonstrated that the model correctly assigned higher weights to frames depicting the core action (e.g., an athlete performing on parallel bars) and lower weights to frames with motion blur or less relevant activity (e.g., an athlete standing on the ground or running up).

Limitations and Future Work

  • Limitations: The paper primarily focuses on trimmed action videos (videos containing a single action). The performance on untrimmed videos with multiple actions or complex temporal structures might be less robust. While the attention mechanism helps, it's still a form of "late fusion" of snippet-level predictions.
  • Future Work: The authors plan to extend their attention model to account for:
  • Spatial attention: Focusing on specific regions within frames, not just temporal segments.
  • Long-term temporal attention: Potentially using Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks for more sophisticated temporal modeling in untrimmed videos.

Summary updated

Abstract

Research in human action recognition has accelerated significantly since the introduction of powerful machine learning tools such as Convolutional Neural Networks (CNNs). However, effective and efficient methods for incorporation of temporal information into CNNs are still being actively explored in the recent literature. Motivated by the popular recurrent attention models in the research area of natural language processing, we propose the Attention-aware Temporal Weighted CNN (ATW CNN) for action recognition in videos, which embeds a visual attention model into a temporal weighted multi-stream CNN. This attention model is simply implemented as temporal weighting yet it effectively boosts the recognition performance of video representations. Besides, each stream in the proposed ATW CNN framework is capable of end-to-end training, with both network parameters and temporal weights optimized by stochastic gradient descent (SGD) with back-propagation. Our experimental results on the UCF-101 and HMDB-51 datasets showed that the proposed attention mechanism contributes substantially to the performance gains with the more discriminative snippets by focusing on more relevant video segments.
BibTeX
@inproceedings{wang2018action,
  author = {Le Wang and Jinliang Zang and Qilin Zhang and Zhenxing Niu and Gang Hua and Nanning Zheng},
  title = {{Action Recognition by an Attention-Aware Temporal Weighted Convolutional Neural Network}},
  booktitle = {Sensors},
  year = {2018},
  url = {https://openreview.net/forum?id=VqqD-o50ZV66},
}
Figure