Action anticipation using latent goal learning

Debaditya Roy, Basura Fernando

Introduction

In computer vision, identifying a current action—action recognition—is a well-established task. However, predicting the future—action anticipation—remains a significant challenge. A recent study from researchers at A*STAR Singapore proposes that the key to accurate anticipation lies in understanding the "latent goals" that drive human behavior.

The Challenge of Action Anticipation

Humans perform sequences of actions directed toward a specific objective. For example, if an individual is observed cracking an egg, their subsequent action depends entirely on their end goal: making pancakes, frying eggs, or preparing a cake.

Conceptual overview of action anticipation: a single observed action can lead to multiple goals and subsequent actions.

Current AI models struggle with this task due to ambiguity in early action sequences, a lack of explicit goal labels in unscripted video data, and the varying temporal scales of human activities. To address these issues, the authors introduce a framework that learns a "latent goal"—a mathematical representation of the final objective derived directly from visual features, without requiring manual goal annotations.

Methodology: Learning Latent Goals

The proposed model treats the goal as a destination within a high-dimensional feature space. The architecture consists of several key components:

The proposed framework architecture, highlighting the Latent Goal Predictor and Action Sampler components.
  1. Latent Goal Predictor: A Stacked RNN processes observed video frames to generate a latent goal representation.
  2. Action Candidate Sampling: An LSTM samples multiple plausible "next action" candidates rather than predicting a single outcome.
  3. Selection Logic: To pick the best candidate, the model applies two novel constraints:
  4. Goal Closeness: The visual state resulting from the next action should be closer to the latent goal than the current state.
  5. Goal Consistency: The latent goal itself should remain stable before and after the action is performed.
  6. Loss Functions: Beyond standard classification loss, the team introduced specific Max-Margin losses for goal closeness and consistency to train the network to respect these logical properties of human behavior.

Key Insights and Experimental Results

The model was evaluated on three major benchmarks: 50Salads, Breakfast, and EPIC-KITCHENS55.

Performance Breakthroughs

On the 50Salads dataset, which features long, goal-oriented sequences, the model reached 59.6% accuracy, shattering the previous state-of-the-art of 40.7%. On the Breakfast dataset, it achieved 47.2% accuracy, slightly edging out previous methods.

Salient Features and Object Focus

The researchers found that max-pooling temporal features outperformed other aggregation methods, likely because it preserves the most salient features of an action that might otherwise be "washed out" by averaging. Additionally, in the unscripted EPIC-KITCHENS environment, the model was significantly better at predicting the noun (the object the person will touch next) than the verb (what they will do with it). This suggests that latent goals are highly tied to the objects required for a task.

Conclusion

By shifting the focus from explicit labels to latent goal learning, this research provides a robust framework for action anticipation. The results demonstrate that enforcing logical consistency and progress toward a goal significantly improves predictive accuracy, moving us closer to AI systems that can proactively understand human intent.

Summary updated

Abstract

To get something done, humans perform a sequence of actions dictated by a goal. So, predicting the next action in the sequence becomes easier once we know the goal that guides the entire activity. We present an action anticipation model that uses goal information in an effective manner. Specifically, we use a latent goal representation as a proxy for the "real goal" of the sequence and use this goal information when predicting the next action. We design a model to compute the latent goal representation from the observed video and use it to predict the next action. We also exploit two properties of goals to propose new losses for training the model. First, the effect of the next action should be closer to the latent goal than the observed action, termed as "goal closeness". Second, the latent goal should remain consistent before and after the execution of the next action which we coined as "goal consistency". Using this technique, we obtain state-of-the-art action anticipation performance on scripted datasets 50Salads and Breakfast that have predefined goals in all their videos. We also evaluate the latent goal-based model on EPIC-KITCHENS55 which is an unscripted dataset with multiple goals being pursued simultaneously. Even though this is not an ideal setup for using latent goals, our model is able to predict the next noun better than existing approaches on both seen and unseen kitchens in the test set. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
BibTeX
@inproceedings{roy2022action,
  author = {Debaditya Roy and Basura Fernando},
  title = {{Action anticipation using latent goal learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00088},
}
Figure