Deep Action- and Context-Aware Sequence Learning for Activity Recognition and Anticipation

Mohammad Sadegh Ali Akbarian, Fatemehsadat Saleh, Basura Fernando, Mathieu Salzmann, Lars Petersson, Lars AnderssonarXiv 2016 · OpenReview

TL;DR

This paper introduces a novel multi-stage LSTM architecture that effectively combines global scene context (context-aware features) with localized action-specific information (action-aware features) for improved video activity recognition and anticipation. By leveraging both types of information, their method achieves state-of-the-art results on standard benchmarks using only RGB frames.

What Problem Does This Paper Address?

Recognizing and anticipating human actions in videos is crucial for many applications, from autonomous driving to surveillance. Existing methods often fall into two camps: those that focus on the global context of a scene (e.g., the entire image or video frame) and those that try to pinpoint the specific regions where an action is happening. The problem is that global context can be misleading (e.g., playing a guitar can happen anywhere), while focusing only on the action might miss helpful contextual cues (e.g., playing soccer usually happens on a grass field). This paper aims to bridge this gap by effectively combining both context-aware and action-aware information.

Methods

The core of the proposed solution is a multi-stage Long Short-Term Memory (LSTM) architecture. This architecture is designed to sequentially process two types of features extracted from video frames:

  • Context-Aware Features: These are derived from a standard VGG-16 network, providing a global representation of the scene. The output of the fc7 layer is used as these features.
  • Action-Aware Features: To extract these, the authors adapt the Class Activation Map (CAM) concept from object detection. They use CAMs generated from the activations of the conv5-3 layer of a fine-tuned CNN. This process highlights regions in the image that are most indicative of a specific action, without requiring explicit bounding box annotations.

The multi-stage LSTM first processes the context-aware features. The output of this first stage is then combined with the action-aware features in a second LSTM stage to produce the final prediction. This sequential processing allows the model to first get a general understanding of the scene and then refine it with localized action information.

A novel loss function (pLGL) is also introduced, designed to encourage earlier and more accurate predictions, which is particularly beneficial for action anticipation. This loss function has a linearly increasing penalty for false positives, acknowledging that early frames might be ambiguous.

Datasets

The paper evaluates its method on two widely used benchmarks:

  • UCF-101: A dataset containing 13,320 videos across 101 action classes, known for its diversity and challenging conditions.
  • JHMDB-21: A dataset of realistic videos from various sources, featuring 928 videos and 21 action classes.

Key Insights and Ideas

  • The Synergy of Context and Action: The most significant insight is that combining global scene context with localized action cues leads to superior performance compared to using either alone. This addresses the limitations of previous methods that were biased towards one type of information.
  • Sequential Processing is Key: The multi-stage LSTM architecture, which processes context-aware features before action-aware features, proves more effective than simply concatenating features or using parallel LSTMs. This suggests a hierarchical approach where general scene understanding informs specific action localization.
  • Actionness Without Annotations: The proposed method for extracting action-aware features using CAMs is notable because it achieves localization capabilities without requiring explicit bounding box or segmentation annotations, making it more practical.
  • Loss Function for Anticipation: The development of a specialized loss function (pLGL) that penalizes false positives more strongly over time is crucial for improving action anticipation, where early predictions are paramount.

Key Results

  • The proposed Deep Action- and Context-Aware Sequence Learning method achieves state-of-the-art accuracy on both UCF-101 (83.3%) and JHMDB-21 (58.41%) using only RGB frames.
  • The multi-stage LSTM architecture consistently outperforms simpler fusion strategies like concatenation or parallel LSTMs.
  • Using context-aware features first in the multi-stage LSTM yields better results than the reverse order.
  • The proposed pLGL loss function leads to higher accuracies compared to standard cross-entropy (CE) and exponentially growing (EGL) losses.
  • The method also shows promise for action anticipation, outperforming other losses in early prediction scenarios.

Limitations and Future Work

  • Optical Flow Integration: While the paper demonstrates that optical flow can be incorporated to further boost performance (achieving 88.7% on UCF-101), it acknowledges that a more integrated architecture for fusing motion-aware features with context- and action-aware ones is a topic for future research.
  • Computational Cost: Although the method is presented as efficient, deep learning models, especially multi-stage ones, can still be computationally intensive.
  • Dataset Specificity: While evaluated on standard datasets, the generalizability to even more diverse and challenging real-world scenarios might require further investigation.

The authors suggest exploring new ways to incorporate additional information sources, such as dense trajectories and human skeletons, in future work.

Summary updated

Abstract

Action recognition and anticipation are key to the success of many computer vision applications. Existing methods can roughly be grouped into those that extract global, context-aware representations of the entire image or sequence, and those that aim at focusing on the regions where the action occurs. While the former may suffer from the fact that context is not always reliable, the latter completely ignore this source of information, which can nonetheless be helpful in many situations. In this paper, we aim at making the best of both worlds by developing an approach that leverages both context-aware and action-aware features. At the core of our method lies a novel multi-stage recurrent architecture that allows us to effectively combine these two sources of information throughout a video. This architecture first exploits the global, context-aware features, and merges the resulting representation with the localized, action-aware ones. Our experiments on standard datasets evidence the benefits of our approach over methods that use each information type separately. We outperform the state-of-the-art methods that, as us, rely only on RGB frames as input for both action recognition and anticipation.
BibTeX
@misc{akbarian2016deep,
  author = {Mohammad Sadegh Ali Akbarian and Fatemehsadat Saleh and Basura Fernando and Mathieu Salzmann and Lars Petersson and Lars Andersson},
  title = {{Deep Action- and Context-Aware Sequence Learning for Activity Recognition and Anticipation}},
  year = {2016},
  eprint = {1611.05520},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.05520},
}
Figure