Looking Ahead: Anticipating Pedestrians Crossing with Future Frames Prediction

Mohamed Chaabane, Ameni Trabelsi, Nathaniel Blanchard, Ross BeveridgeWACV 2020

TL;DR

This paper introduces an end-to-end deep learning model that predicts future video frames to anticipate pedestrian crossing actions, aiming to improve autonomous vehicle safety. By combining future frame prediction with action recognition, the model achieves state-of-the-art performance on the challenging JAAD dataset, offering a crucial step towards more human-like defensive driving.

What Problem Does This Paper Address?

Self-driving cars need to navigate complex urban environments, where predicting the intentions and actions of vulnerable road users like pedestrians is paramount for safety. Human drivers intuitively anticipate pedestrian behavior through subtle cues and context. Current autonomous systems often struggle to replicate this nuanced understanding, leading to potential accidents if pedestrian actions are mispredicted. Specifically, the challenge lies in accurately forecasting whether a pedestrian will step into the vehicle's path, especially in dynamic and unpredictable scenarios. This paper tackles this by developing a model that not only predicts future visual information but also uses it to infer pedestrian intent.

Methods

The proposed solution is an end-to-end, two-stage model:

  1. Future Frames Prediction: This stage uses a self-supervised encoder-decoder network. The encoder processes a sequence of past video frames (N frames) using 3D convolutional layers to capture spatio-temporal features. The decoder then reconstructs these features to predict the next N future frames. Key architectural choices include depth-wise separable convolutional LSTMs for efficiency and lateral connections to preserve fine-grained detail from the input frames.
  2. Pedestrian Action Prediction: The predicted future frames from the first stage are fed into a fine-tuned action recognition network (Temporal 3D ConvNets - T3D). This network is trained to classify whether the pedestrian will cross the street. The final classification layer is adapted to output a single probability.

The entire model is trained end-to-end using a multi-task learning objective that balances the loss from future frame prediction and the cross-entropy loss for action classification.

Datasets

The primary dataset used for training and evaluation is the Joint Attention for Autonomous Driving (JAAD) dataset [26]. This dataset is specifically designed for studying pedestrian and driver behavior in complex urban scenarios, featuring approximately 240 hours of driving videos with a focus on pedestrian crossing events. It offers a variety of conditions, including weather, lighting, and interactions with other vehicles. The authors also mention pre-training the future frames prediction component on the Moving MNIST dataset and the action recognition component on the UCF101 dataset.

Key Insights and Ideas

  • Predicting the Future is Key to Anticipation: The core idea is that accurately predicting what the scene will look like in the near future provides a richer context for understanding current actions and predicting future intentions, especially for pedestrians.
  • Bridging the Gap Between Perception and Prediction: By directly linking future frame prediction to pedestrian action classification, the model moves beyond static scene analysis to dynamic, predictive behavior modeling.
  • Importance of Spatio-Temporal Detail: The use of 3D convolutions and lateral connections in the encoder-decoder is crucial for capturing both spatial appearance and temporal evolution, leading to more realistic future frame predictions.
  • Efficiency through Depth-wise Separable Convolutions: Employing depth-wise separable convLSTMs in the decoder significantly speeds up the model without a substantial drop in performance, making it more practical for real-time applications.

Key Results

  • State-of-the-Art Pedestrian Crossing Prediction: The model achieves an Average Precision (AP) of 86.7% on the JAAD dataset, outperforming previous state-of-the-art methods by a significant margin (e.g., 81.14% AP for Res-EnDec [14]).
  • Improved Future Frame Prediction: The model demonstrates superior performance in future frame prediction, achieving a pixel-wise L1 error of 1.12 ± 0.32 (×10⁻¹), which is lower than competing methods like Res-EnDec [14], PredRNN++ [42], and PredNet [24].
  • Effective Look-Ahead Time: The model provides a predictive capability with an effective look-ahead time of 430 ms, allowing autonomous vehicles sufficient time for defensive reactions.
  • Model Efficiency: The proposed model achieves a running time of 102.48 ms for processing 16 frames, making it suitable for real-time deployment.
  • Ablation Study Validation: An ablation study confirmed the effectiveness of individual components, such as depth-wise separable convLSTMs, residual connections, and lateral connections, in improving performance and efficiency.

Limitations and Future Work

  • Limitations: While the model performs well, the paper acknowledges that the quality of predicted frames degrades over time, especially beyond the initial few frames. The model's performance is also evaluated on specific splits of the JAAD dataset, and its robustness across even more diverse and challenging real-world scenarios could be further tested.
  • Future Work: The authors suggest that their future frame prediction model could be extended to predict the behavior of other agents, such as vehicles. They also propose exploring the prediction of pedestrian intent (rather than just action) by leveraging additional labels from the JAAD dataset. The ultimate goal remains to design a fully autonomous system that mimics the defensive and anticipatory behavior of human drivers.

Summary updated

BibTeX
@inproceedings{chaabane2020looking,
  author = {Mohamed Chaabane and Ameni Trabelsi and Nathaniel Blanchard and Ross Beveridge},
  title = {{Looking Ahead: Anticipating Pedestrians Crossing with Future Frames Prediction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure