From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation

Xin Liu, Chao Hao, Zitong Yu, Huanjing Yue, Jingyu Yang

TL;DR

This paper introduces ARR, a novel end-to-end architecture for action anticipation that decomposes the task into action recognition and sequence reasoning. By leveraging attention mechanisms and a novel Next Action Prediction (NAP) task, ARR effectively models the statistical relationships between actions, outperforming previous methods on several benchmark datasets. The work also proposes an unsupervised pre-training method to address the data scarcity challenge in action anticipation.

What Problem Does This Paper Address?

Predicting future human actions from video, known as action anticipation, is a crucial task for AI systems in various applications, from smart homes to autonomous driving. While action recognition focuses on identifying actions happening now, anticipation aims to predict what will happen next. This is significantly more challenging because it requires understanding the temporal dependencies and correlations between actions. Previous methods often treated action anticipation as a direct prediction task, failing to effectively leverage the inherent sequential nature of human behavior and the rich prior knowledge embedded in action sequences. This paper addresses the gap by proposing a method that explicitly models these action-to-action relationships.

Methods

The core of the proposed method, ARR (Anticipation via Recognition and Reasoning), is a two-stage architecture:

  • Action Recognition: The first stage uses a vision Transformer-based encoder (specifically AIM) to process input video clips and extract features representing the current action. This stage is supervised by ground truth action labels.
  • Sequence Reasoning: The second stage employs a causal decoder, inspired by large language models, to perform sequence reasoning. This decoder takes the sequence of action features from the recognition stage and predicts the next action. The key innovation here is the Next Action Prediction (NAP) task, which is analogous to Next Token Prediction (NTP) in NLP. NAP explicitly learns the statistical relationships between consecutive actions.
  • Unsupervised Pre-training: To overcome the need for large amounts of labeled data for the sequence reasoning component, the authors propose an unsupervised pre-training method. This method leverages the natural temporal order of videos to train the decoder, enhancing its reasoning capabilities without requiring explicit action annotations.

Datasets

The authors evaluated their method on three widely used action anticipation datasets:

  • Epic-Kitchens-100 (EK100): A large-scale egocentric action dataset with 9K action segments, 97 verb classes, and 300 noun classes.
  • EGTEA Gaze+: An egocentric video dataset with 10K action annotations, covering 19 verbs, 51 nouns, and 106 unique actions.
  • 50salads: A smaller-scale third-person perspective video dataset with 0.9K action segments and 17 different actions.

Key Insights and Ideas

  • Action sequences hold valuable prior knowledge: The paper strongly emphasizes that actions are not isolated events but rather form correlated sequences. Understanding these correlations is crucial for accurate anticipation.
  • Decomposition is key: Breaking down the complex action anticipation task into more manageable sub-tasks (recognition and reasoning) allows for more effective modeling.
  • Leveraging LLM architectures for sequence reasoning: The adoption of causal decoders, proven effective in large language models for sequence generation, is a smart transfer of technology to the action anticipation domain.
  • Unsupervised pre-training is vital for data-scarce domains: The proposed unsupervised pre-training method effectively addresses the common challenge of limited labeled data in action anticipation, significantly improving performance.
  • Attention mechanisms are powerful for spatio-temporal understanding: The use of attention in both the encoder and decoder allows the model to focus on relevant parts of the video and temporal sequences.

Key Results

  • State-of-the-art performance: ARR achieved competitive results and often surpassed previous state-of-the-art methods on the EK100, EGTEA Gaze+, and 50salads datasets. For instance, on EK100, ARR achieved the best performance among single-modality methods, outperforming AVT.
  • Effectiveness of unsupervised pre-training: Ablation studies showed that the proposed unsupervised pre-training method significantly improved the performance of the causal decoder compared to random initialization or even a pre-trained GPT-2 decoder, demonstrating its effectiveness.
  • Robustness to anticipation time: The method showed good performance across different anticipation time settings on EGTEA Gaze+.
  • Generality: The success on both egocentric (EK100, EGTEA Gaze+) and third-person (50salads) datasets demonstrates the generality of the proposed approach.
  • Attention visualization: Visualizations confirmed that ARR's attention mechanism effectively focuses on regions relevant to the current action, leading to more precise and specific focus areas.

Limitations and Future Work

  • Action density and speed: The paper acknowledges that when actions are very dense or the speed of actions is not considered, ARR can make incorrect predictions (e.g., predicting "eat bread" instead of "close ketchup"). This suggests that incorporating temporal dynamics beyond just sequential order could be beneficial.
  • Domain gap for text-based pre-training: While comparing to GPT-2, the authors noted that its performance might be limited by the domain gap between text pre-training and video pre-training. This highlights the advantage of video-specific unsupervised pre-training.
  • Future work: The authors express interest in further exploring unsupervised training methods for action anticipation. They also implicitly suggest that improving the handling of action speed and density could lead to more robust predictions.

Summary updated

Abstract

The action anticipation task refers to predicting what action will happen based on observed videos, which requires the model to have a strong ability to summarize the present and then reason about the future. Experience and common sense suggest that there is a significant correlation between different actions, which provides valuable prior knowledge for the action anticipation task. However, previous methods have not effectively modeled this underlying statistical relationship. To address this issue, we propose a novel end-to-end video modeling architecture that utilizes attention mechanisms, named Anticipation via Recognition and Reasoning (ARR). ARR decomposes the action anticipation task into action recognition and sequence reasoning tasks, and effectively learns the statistical relationship between actions by next action prediction (NAP). In comparison to existing temporal aggregation strategies, ARR is able to extract more effective features from observable videos to make more reasonable predictions. In addition, to address the challenge of relationship modeling that requires extensive training data, we propose an innovative approach for the unsupervised pre-training of the decoder, which leverages the inherent temporal dynamics of video to enhance the reasoning capabilities of the network. Extensive experiments on the Epic-kitchen-100, EGTEA Gaze+, and 50salads datasets demonstrate the efficacy of the proposed methods. The code is available at https://github.com/linuxsino/ARR.
BibTeX
@article{liu2024from,
  author = {Xin Liu and Chao Hao and Zitong Yu and Huanjing Yue and Jingyu Yang},
  title = {{From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation}},
  journal = {ACM Trans. Multim. Comput. Commun. Appl.},
  year = {2024},
  url = {https://openreview.net/forum?id=Nsfd8cp0PY},
}
Figure