Early Action Prediction with Generative Adversarial Networks

Dong Wang, Yuan Yuan, Qi Wang

Abstract

Action Prediction is aimed to determine what action is occurring in a video as early as possible, which is crucial to many online applications, such as predicting a traffic accident before it happens and detecting malicious actions in the monitoring system. In this work, we address this problem by developing an end-to-end architecture that improves the discriminability of features of partially observed videos by assimilating them to features from complete videos. For this purpose, the generative adversarial network is introduced for tackling action prediction problem, which improves the recognition accuracy of partially observed videos though narrowing the feature difference of partially observed videos from complete ones. Specifically, its generator comprises of two networks: a CNN for feature extraction and an LSTM for estimating residual error between features of the partially observed videos and complete ones, and then the features from CNN adds the residual error from LSTM, which is regarded as the enhanced feature to fool a competing discriminator. Meanwhile, the generator is trained with an additional perceptual objective, which forces the enhanced features of partially observed videos are discriminative enough for action prediction. Extensive experimental results on UCF101, BIT and UT-Interaction datasets demonstrate that our approach outperforms the state-of-the-art methods, especially for videos that less than 50% portion of frames is observed.

Introduction

In computer vision, action recognition identifies an action after a video concludes. However, action prediction—identifying an action as early as possible while it is still occurring—presents a much greater challenge. The primary difficulty lies in initial ambiguity; for instance, the beginning of a "running" sequence and a "triple jump" may appear identical.

Standard models often struggle with partial videos because they contain significantly less information than complete ones. This research addresses this "information gap" by teaching a model to "hallucinate" the missing features of a complete video based on initial frames. By assimilating partial features to complete ones, the model learns the temporal evolution of an action more effectively.

Overview of the GAN-based feature enhancement for early action prediction.

Methodology

The proposed framework utilizes a novel GAN-based architecture designed to enhance the features of partially observed videos. The system integrates three primary components:

  1. The Generator: This component produces "enhanced features" rather than images. It employs a CNN for spatial feature extraction and an LSTM network to estimate the "residual error"—the difference between partial features and those of a hypothetical complete video. This residual learning approach is computationally efficient and focuses the network on missing temporal information.
  2. The Discriminator: Acting as a critic, the discriminator attempts to distinguish between "real" features (from complete videos) and "fake" enhanced features. This competition drives the generator to produce features as informative as those from full-length clips.
  3. The Perceptual Network: To ensure the enhanced features are not just realistic but also discriminative for classification, a perceptual network provides feedback based on action prediction accuracy.
Detailed architecture showing the Generator, Perceptual, and Discriminator networks.

Experimental Results

The researchers evaluated the approach on three benchmarks: UCF101 (101 action categories), BIT-Interaction (human-to-human interactions), and UT-Interaction (surveillance-based interactions). The proposed method consistently outperformed state-of-the-art (SOTA) models:

  • UCF101: Achieved an average accuracy of 87.7% (compared to 81.3% for previous SOTA). Notably, at only 10% observation, the model improved accuracy by 27.8% over existing methods.
  • BIT-Interaction: Reached 97.5% accuracy at 50% video completion.
  • Early Prediction Success: On UCF101, 36 out of 101 actions achieved over 90% accuracy after seeing just the first 10% of the video.

Conclusion

This work demonstrates that GANs can effectively bridge the information gap in early action prediction by assimilating partial video features to complete ones. While performance remains challenging at extremely low observation ratios (10%) for certain subtle interactions, the significant accuracy gains at early stages suggest the model successfully captures priors of human movement. Future research may explore integrating additional modalities like audio to further reduce early-stage ambiguity.

Summary updated

Abstract

Action Prediction is aimed to determine what action is occurring in a video as early as possible, which is crucial to many online applications, such as predicting a traffic accident before it happens and detecting malicious actions in the monitoring system. In this work, we address this problem by developing an end-to-end architecture that improves the discriminability of features of partially observed videos by assimilating them to features from complete videos. For this purpose, the generative adversarial network is introduced for tackling action prediction problem, which improves the recognition accuracy of partially observed videos though narrowing the feature difference of partially observed videos from complete ones. Specifically, its generator comprises of two networks: a CNN for feature extraction and an LSTM for estimating residual error between features of the partially observed videos and complete ones, and then the features from CNN adds the residual error from LSTM, which is regarded as the enhanced feature to fool a competing discriminator. Meanwhile, the generator is trained with an additional perceptual objective, which forces the enhanced features of partially observed videos are discriminative enough for action prediction. Extensive experimental results on UCF101, BIT and UT-Interaction datasets demonstrate that our approach outperforms the state-of-the-art methods, especially for videos that less than 50% portion of frames is observed.
BibTeX
@inproceedings{wang2019early,
  author = {Dong Wang and Yuan Yuan and Qi Wang},
  title = {{Early Action Prediction with Generative Adversarial Networks}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=MNhKRN3Cb8u},
}
Figure