Early Action Prediction With Generative Adversarial Networks

Dong Wang, Yuan Yuan, Qi Wang

TL;DR

Researchers have developed a new end-to-end framework that uses Generative Adversarial Networks (GANs) to predict video actions (like jumping or hugging) long before they are completed. By "hallucinating" the missing information in partial video clips to make them look like features from full-length videos, the model significantly improves accuracy, especially when only the first 10-30% of an action is visible.

What Problem Does This Paper Address?

In computer vision, action recognition is a well-studied field where a model identifies what happened after a video ends. However, action prediction is much harder. It requires the system to identify the action as early as possible—ideally while it is still happening.

The core challenge is ambiguity. At the very beginning of a video, "running" and a "triple jump" look identical. A system only sees the difference once the athlete actually takes the leap. Because partial videos contain less information than complete ones, standard models often struggle to find discriminative features early on. This paper aims to bridge that "information gap" by teaching a model to imagine the features of a complete video based only on a few initial frames.

Methods

The authors propose a novel GAN-based architecture designed to enhance the features of partially observed videos. The system consists of three main components:

  1. The Generator: Unlike a standard GAN that generates images, this generator produces "enhanced features." It uses a CNN for spatial feature extraction and an LSTM (Long Short-Term Memory) network to estimate the "residual error"—essentially the difference between the current partial features and the features of a hypothetical complete video.
  2. The Discriminator: This component acts as a critic. It tries to distinguish between "real" features (taken from actual complete videos) and "fake" features (the enhanced ones produced by the generator). This competition forces the generator to produce features that are as rich and informative as those from full-length clips.
  3. The Perceptual Network: To ensure the generator doesn't just make "realistic" features but actually useful ones, a perceptual network is added. This network is trained to classify the action. If the generator produces features that fool the discriminator but lead to the wrong classification, the perceptual network provides feedback to correct it.

The model also utilizes a Two-Stream approach, processing both standard RGB frames and "Optical Flow" (which captures motion) to maximize predictive power.

Datasets

The researchers evaluated their approach on three widely recognized benchmarks:

  • UCF101: A diverse dataset of 101 action categories (sports, playing instruments, etc.).
  • BIT-Interaction: Focused on eight classes of human-to-human interactions (hugging, shaking hands, etc.).
  • UT-Interaction: A dataset of human interactions captured in real-world surveillance settings.

Key Insights and Ideas

  • Feature Assimilation: The most powerful idea here is that we shouldn't just train on partial data; we should try to make partial data look like full data. By "assimilating" partial features to complete ones, the model learns the temporal evolution of an action.
  • Residual Learning for Time: Instead of trying to predict the future frames directly, the model predicts the residual error of the features. This is computationally more efficient and focuses the network on what is missing from the current observation.
  • The "Instant" Prediction Jump: The model shows a massive performance spike at very low observation ratios (10%). This suggests that the GAN is successfully capturing the "priors" of human movement—knowing that a certain stance likely leads to a certain action.

Key Results

The proposed method consistently outperformed state-of-the-art (SOTA) models across all datasets:

  • UCF101: Achieved an average accuracy of 87.7% (vs. 81.3% for the previous SOTA). Most impressively, at only 10% observation, the model improved accuracy by 27.8% over previous methods.
  • BIT-Interaction: Reached 97.5% accuracy when the video was 50% complete.
  • Early Success: On UCF101, 36 out of 101 actions achieved over 90% prediction accuracy after seeing just the first segment (10%) of the video.

Limitations and Future Work

While the model is highly effective, the authors noted that some datasets, like BIT-Interaction, remain difficult at the 10% mark. This is because many human interactions (like a handshake) begin with two people simply standing near each other, which provides almost zero discriminative information regardless of the model's power.

Future work could involve exploring more complex temporal modeling or integrating other modalities (like audio) to further reduce ambiguity in those first few frames. Additionally, while the discriminator is used during training, it is discarded during inference to save on computation—exploring ways to use the discriminator's "uncertainty" during real-time prediction could be a promising path.

Summary updated

Abstract

Action Prediction is aimed to determine what action is occurring in a video as early as possible, which is crucial to many online applications, such as predicting a traffic accident before it happens and detecting malicious actions in the monitoring system. In this paper, we address this problem by developing an end-to-end architecture that improves the discriminability of features of partially observed videos by assimilating them to features from complete videos. For this purpose, the generative adversarial network is introduced for tackling action prediction problem, which improves the recognition accuracy of partially observed videos though narrowing the feature difference of partially observed videos from complete ones. Specifically, its generator comprises of two networks: a CNN for feature extraction and an LSTM for estimating residual error between features of the partially observed videos and complete ones, and then the features from CNN adds the residual error from LSTM, which is regarded as the enhanced feature to fool a competing discriminator. Meanwhile, the generator is trained with an additional perceptual objective, which forces the enhanced features of partially observed videos are discriminative enough for action prediction. The extensive experimental results on UCF101, BIT, and UT-Interaction datasets demonstrate that our approach outperforms the state-of-the-art methods, especially for videos that less than 50% portion of frames is observed.
BibTeX
@inproceedings{wang2019early,
  author = {Dong Wang and Yuan Yuan and Qi Wang},
  title = {{Early Action Prediction With Generative Adversarial Networks}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=MNhKRN3Cb8u},
}

Loading related content...

Figure