Anticipating Future Relations via Graph Growing for Action Prediction

Xinxiao Wu, Jianwei Zhao, Ruiqi WangAAAI 2021 · OpenReview

Abstract

Predicting actions from partially observed videos is challenging as the partial videos containing incomplete action executions have insufficient discriminative information for classification. Recent progress has been made through enriching the features of the observed video part or generating the features for the unobserved video part, but without explicitly modeling the fine-grained evolution of visual object relations over both space and time. In this paper, we investigate how the interaction and correlation between visual objects evolve and propose a graph growing method to anticipate future object relations from limited video observations for reliable action prediction. There are two tasks in our method. First, we work with spatial-temporal graph neural networks to reason object relations in the observed video part. Then, we synthesize the spatial-temporal relation representation for the unobserved video part via graph node generation and aggregation. These two tasks are jointly learned to enable the anticipated future relation representation informative to action prediction. Experimental results on two action video datasets demonstrate the effectiveness of our method.

Introduction

Action prediction is the difficult cousin of action recognition. While recognition occurs after a video concludes, prediction requires a system to guess the action while it is still unfolding. This is incredibly challenging because partial videos often lack the "smoking gun" frames that make an action obvious—for instance, the moment a ball actually enters a hoop.

Most existing methods attempt to solve this by hallucinating missing visual features or transferring knowledge from full videos to partial ones. However, they often ignore the fine-grained structural evolution of the scene—specifically, how objects (like a hand and a cup) interact over time. Humans naturally fill in these gaps using intuition, a concept in psychology called the Gestalt law. This paper proposes a way for AI to do the same by treating video sequences as evolving graphs.

Methodology: The Graph Growing Framework

The proposed framework consists of two primary stages: Relation Reasoning and Relation Synthesizing.

Relation Reasoning (The "Now")

In the observed portion of the video, the model performs both spatial and temporal reasoning:

  • Spatial Reasoning: For each frame, the model builds a graph where nodes represent objects (detected via Faster R-CNN) and edges represent their relationships. A Gated Graph Neural Network (GGNN) allows these nodes to interact and refine their representations.
  • Temporal Reasoning: To capture how relations change over time, a Long Short-Term Graph Convolutional Network (LST-GCN) is introduced. This captures both rapid movements and long-term trends by analyzing the video at multiple temporal scales.

Relation Synthesizing (The "Future")

To anticipate the unobserved future, the model takes the observed sub-graph and uses a graph auto-encoder to "grow" new nodes. This process is guided by Local and Global Alignment Losses, ensuring the predicted future relations are realistic continuations of the observed ones and that the final "complete" graph matches the structure of a ground-truth full video.

The overall framework of the proposed graph growing method for action prediction.

Experiments and Results

The model was evaluated on two major datasets: UCF101 (general actions) and 20BN-something-something (v1) (fine-grained human-object interactions). The latter is particularly difficult because actions are highly dependent on temporal structure.

Key Findings

  • Early Accuracy: On the UCF101 dataset, the model achieved 75.85% accuracy when seeing only 10% of the video, outperforming the previous best method by over 7%.
  • Fine-Grained Success: On the "something-something" dataset, the model maintained a consistent lead over competitors like MSRNN and mem-LSTM at every stage of the video observation ratio.
  • Structure Over Pixels: By predicting future relational structures rather than raw pixels, the model captures the "essence" of an action more efficiently and with less noise.
Qualitative results comparing the proposed method against a reasoning-only baseline across different observation ratios.

Visualizations demonstrate that while reasoning-only models often get stuck on incorrect guesses (e.g., confusing "turning something upside down" with "picking something up"), the graph-growing model corrects itself much earlier by anticipating the logical conclusion of the movement.

Conclusion

The "Graph Growing" method demonstrates that modeling the evolution of object relations is a powerful tool for action prediction. While the method currently relies on the quality of initial object detectors, it marks a significant step toward more "intuitive" AI that can perceive incomplete patterns as a whole. Future work may explore incorporating semantic knowledge to understand not just where objects are, but how they are intended to be used.

Summary updated

Abstract

Predicting actions from partially observed videos is challenging as the partial videos containing incomplete action executions have insufficient discriminative information for classification. Recent progress has been made through enriching the features of the observed video part or generating the features for the unobserved video part, but without explicitly modeling the fine-grained evolution of visual object relations over both space and time. In this paper, we investigate how the interaction and correlation between visual objects evolve and propose a graph growing method to anticipate future object relations from limited video observations for reliable action prediction. There are two tasks in our method. First, we work with spatial-temporal graph neural networks to reason object relations in the observed video part. Then, we synthesize the spatial-temporal relation representation for the unobserved video part via graph node generation and aggregation. These two tasks are jointly learned to enable the anticipated future relation representation informative to action prediction. Experimental results on two action video datasets demonstrate the effectiveness of our method.
BibTeX
@inproceedings{wu2021anticipating,
  author = {Xinxiao Wu and Jianwei Zhao and Ruiqi Wang},
  title = {{Anticipating Future Relations via Graph Growing for Action Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=H6kPbV2xm-},
}
Figure