DBDNet: Learning Bi-directional Dynamics for Early Action Prediction

Guoliang Pang, Xionghui Wang, Jianfang Hu, Qing Zhang, Wei-Shi Zheng

Introduction

Early action prediction aims to identify a human activity before it is completed. This is critical for time-sensitive applications like visual surveillance or medical monitoring (e.g., fall detection). The primary challenge lies in the inherent uncertainty of partial video sequences, which lack the full context of the action. Traditional methods often rely on unidirectional temporal dynamics (past-to-future), which can be noisy and insufficient for robust prediction.

Methodology: Bi-directional Dynamics Network (DBDNet)

The authors propose DBDNet, an encoder-decoder architecture that leverages the bi-directional symmetry of human actions. The core hypothesis is that actions are recognizable whether played forward or backward, and combining these perspectives provides a richer context for characterizing motion.

Figure 4: Illustration of bi-directional temporal dynamics in an action sequence. The model learns to synthesize the future (forward) and reason back to the past (backward) to create a robust representation.

The network consists of three main components:

  1. Motion Synthesis Block (Past-to-Future): Utilizes an Auto-Conditioned LSTM (acLSTM) to generate future motion descriptors from observed frames. The acLSTM's self-conditioning during training helps mitigate error accumulation in long-term generation.
  2. Motion Reasoning Block (Future-back-to-Past): This key innovation attempts to reconstruct the original observed motions in reverse order from the synthesized future. This serves as a logical consistency check; if the predicted future cannot reconstruct the past, the prediction is likely flawed.
  3. Action Prediction Block: Features from both blocks are fused and processed by a Bi-directional LSTM (Bi-LSTM) for final classification.

Experimental Results

DBDNet was evaluated on two benchmark datasets: UCF101 (13,320 videos, 101 categories) and NTU RGB+D (56,000+ videos, 60 categories). The latter is particularly challenging due to visually similar starting frames for different actions, such as "drinking water" versus "brushing teeth."

Key findings include:

  • Superior Early Prediction: On UCF101, with only 10% of the video observed, DBDNet achieved 82.67% accuracy, significantly outperforming the previous state-of-the-art (MSRNN) at 68.00%.
  • Robustness on Large Datasets: On NTU RGB+D, DBDNet reached an Area Under the Curve (AUC) of 61.07%, a ~14% improvement over MSRNN (46.56%).
  • Ablation Validation: Removing the Reasoning Block led to a performance drop across all observation ratios, confirming the value of bi-directional learning.

Discussion and Conclusion

The success of DBDNet stems from treating bi-directional symmetry as a form of context. The Reasoning Block acts as a powerful regularizer, ensuring that synthesized futures are logically grounded in the observed past.

However, limitations remain. The model does not explicitly handle the multi-modal nature of future trajectories—where one start could lead to multiple valid endings. Additionally, the computational cost of synthesis and reasoning may impact real-time performance, a critical factor for "early" prediction. Future work could involve tighter integration of feature extraction with bi-directional dynamics and exploring multi-modal future modeling.

Summary updated

Abstract

Predicting future actions from observed partial videos is very challenging as the missing future is uncertain and sometimes has multiple possibilities. To obtain a reliable future estimation, a novel encoder-decoder architecture is proposed for integrating the tasks of synthesizing future motions from observed videos and reconstructing observed motions from synthesized future motions in an unified framework, which can capture the bi-directional dynamics depicted in partial videos along the temporal (past-to-future) direction and reverse chronological (future-back-to-past) direction. We then employ a bi-directional long short-term memory (Bi-LSTM) architecture to exploit the learned bi-directional dynamics for predicting early actions. Our experiments on two benchmark action datasets show that learning bi-directional dynamics benefits the early action prediction and our system clearly outperforms the state-of-the-art methods.
BibTeX
@inproceedings{pang2019dbdnet,
  author = {Guoliang Pang and Xionghui Wang and Jianfang Hu and Qing Zhang and Wei-Shi Zheng},
  title = {{DBDNet: Learning Bi-directional Dynamics for Early Action Prediction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/126},
}
Figure