From Recognition to Prediction: Analysis of Human Action and Trajectory Prediction in Video

Junwei Liang2022 · DOI

TL;DR

This paper tackles the challenging problem of predicting human future trajectories in urban traffic scenes by jointly analyzing human actions and scene semantics. The authors introduce novel models and datasets that achieve state-of-the-art performance, paving the way for safer autonomous driving and robotics applications.

What Problem Does This Paper Address?

Predicting where humans will move next is crucial for applications like autonomous driving and socially-aware robots. However, accurately forecasting human trajectories is difficult due to the complex and dynamic nature of urban environments, where scene semantics and human intent are hard to model. Existing methods often struggle to incorporate high-level semantic attributes and generalize to unseen scenarios. This paper aims to build a robust trajectory prediction system that deeply understands semantic context and human behavior.

Methods

The paper is structured into three parts, each addressing a key aspect of the problem:

  • Part I: Human Action Analysis: This part focuses on developing an efficient object detection and tracking system for videos, similar to those used in self-driving cars. It also tackles action recognition under weakly-supervised learning settings, proposing a method for learning viewpoint-invariant representations to improve generalization.
  • Part II: Pedestrian Trajectory Prediction with Scene Semantics: Here, the authors address trajectory forecasting by incorporating scene semantic understanding. They introduce a "Multiverse" model for multi-future trajectory prediction and a novel "Forking Paths" dataset generated using 3D simulation to better model uncertainty. They also explore learning from 3D simulation data to build robust models.
  • Part III: Joint Analysis of Human Actions and Trajectory Prediction: The final part combines findings from the previous two to develop a robust model for long-term trajectory prediction. This involves jointly predicting short-term trajectories and activities on common benchmarks, and introducing a new human-annotated long-term trajectory and action prediction benchmark with multi-view camera data in urban traffic scenes.

Datasets

  • ActEV: Used for action recognition and detection.
  • VIRAT/ActEV: Used for trajectory prediction.
  • Forking Paths Dataset: A novel, human-annotated synthetic dataset for multi-future trajectory prediction, generated using the CARLA simulator and anchored in real-world data. It aims to address the limitation of existing datasets that only provide one observed future trajectory.
  • MEVA-Trajectory Dataset: A new human-annotated long-term trajectory prediction dataset with multi-view camera data in urban traffic scenes, derived from the MEVA dataset. It features extended length, high-resolution videos with clear human intents and person-vehicle interactions.

Key Insights and Ideas

  • Jointly modeling action and trajectory is crucial: The paper demonstrates that understanding human actions and intentions significantly improves trajectory prediction accuracy, especially for longer horizons.
  • Scene semantics are vital for robust prediction: Incorporating scene understanding helps models generalize better and infer human intent, particularly in complex urban environments.
  • Multi-future prediction is essential for handling uncertainty: The "Multiverse" model and the "Forking Paths" dataset highlight the importance of predicting multiple plausible futures rather than a single deterministic path.
  • 3D simulation can bridge the domain gap: Training on synthetic data generated from 3D simulators, augmented with techniques like SimAug, can lead to models that generalize well to unseen real-world cameras.
  • Weakly-supervised learning unlocks large-scale data: Leveraging webly-labeled data with multi-modal information is key to overcoming the scarcity of manually annotated datasets for action recognition.

Key Results

  • State-of-the-art performance: The proposed models achieve state-of-the-art results on various benchmarks for action recognition, trajectory prediction, and joint prediction tasks.
  • Significant improvements in long-term prediction: The MEVA-Trajectory dataset and associated models demonstrate substantial progress in predicting trajectories over longer horizons (up to 12 seconds).
  • Robustness to unseen data: The SimAug approach shows strong generalization capabilities on unseen cameras and environments, trained solely on synthetic data.
  • Improved action recognition: Weakly-supervised learning with multi-modal data leads to competitive action recognition performance, even outperforming models trained on manually annotated data in some cases.
  • Impact in Academia and Industry: The work has been published in top conferences and journals, received significant citations, and has been implemented in real-world applications like a COVID-19 social distancing system.

Limitations and Future Work

  • Dataset limitations: Existing datasets, even the newly introduced ones, may not fully capture critical corner cases (e.g., person-vehicle negotiation, sudden turns) important for safety-critical applications. More diverse datasets with rich multi-agent interactions are needed.
  • Training schemes: The current models primarily use positive examples for training and do not explicitly penalize unreasonable predictions. Exploring datasets with multiple positive samples and incorporating physical constraints from 3D simulators could improve generalization.
  • Scene semantics utilization: While scene semantics are incorporated, there's room for improvement in how models fully capture correlations between scene classes and trajectories, potentially by revisiting simpler models like Hidden Markov Models.
  • First-person view prediction: Trajectory prediction from first-person view videos, which often involve ego-motions, remains an underexplored area.
  • Computation-accuracy trade-off: A systematic study is needed to balance computational cost and accuracy for real-world deployment, especially on resource-constrained platforms.
  • Unifying vehicle and pedestrian prediction: Current systems often treat vehicle and pedestrian trajectory prediction separately. A unified model could leverage their interactions for better efficiency and accuracy.
  • Common sense reasoning: Incorporating high-level scene attributes and common sense reasoning (e.g., understanding school zones or stadium events) could significantly enhance long-term prediction.
  • Temporal transformation in STAN: The STAN model could be further developed to learn more meaningful temporal transformations for action recognition.

Summary updated

BibTeX
@misc{liang2022from,
  author = {Junwei Liang},
  title = {{From Recognition to Prediction: Analysis of Human Action and Trajectory Prediction in Video}},
  year = {2022},
  doi = {10.1184/r1/21692891.v1},
}
Figure