Prediction of Pedestrian Crossing Behavior Based on Surveillance Video

Xiao Zhou, Hongyu Ren, Tingting Zhang, Xingang Mou, Yi He, Ching-Yao Chan2022 · DOI

TL;DR

This paper tackles the crucial problem of predicting pedestrian crossing behavior from surveillance video, a vital component for autonomous driving safety. They introduce a novel spatio-temporal feature fusion network that effectively combines pedestrian posture, local context, and global context information, achieving state-of-the-art results on a custom-annotated dataset.

What Problem Does This Paper Address?

Ensuring the safety of autonomous vehicles (AVs) hinges on their ability to accurately predict the intentions and actions of vulnerable road users, particularly pedestrians. While vehicle-mounted cameras are common for this task, their limited field of view can be obstructed, leading to missed critical information. This paper proposes using surveillance video, which offers a broader perspective, to predict pedestrian crossing behavior. This is especially valuable for accident-prone areas or critical road sections, providing supplementary data for AV decision-making and reducing accident risks. The core challenge is to effectively fuse diverse information sources – pedestrian's own state, immediate surroundings, and broader traffic context – to make accurate predictions.

Methods

The authors propose a novel "spatio-temporal feature fusion network" that integrates three key types of features:

  • Pedestrian Posture Features: This includes body and head posture, extracted from key points. These are fed into a GRU (Gated Recurrent Unit) for sequential processing.
  • Local Context Features: These capture the immediate environment around the pedestrian, extracted using a pre-trained VGG19 network.
  • Global Context Features: This encompasses broader scene information, including original image features and optical flow features to capture motion.

A key innovation is the use of a "cross-stacked gated recurrence unit (GRU)" structure. This allows for a deeper fusion of local and global context features by interleaving GRU layers and attention mechanisms. The network is designed to recursively process these features, allowing for a more comprehensive understanding of the scene and pedestrian intent. The final output is a binary classification indicating whether the pedestrian will cross the street.

Datasets

The primary dataset used in this research is derived from surveillance video footage recorded on a busy street at the University of California, Berkeley. This dataset was specifically annotated for pedestrian crossing behavior prediction.

  • UC Berkeley Surveillance Dataset: Contains 300 video clips, each featuring a target pedestrian. The videos are approximately 20 seconds long at 15 frames per second, with a resolution of 1920x1080. The dataset includes annotations for pedestrian pose key points and the pedestrian's crossing behavior (crossing or not crossing).

Key Insights and Ideas

  • Surveillance Video Advantage: The paper highlights that surveillance video offers a wider perspective and richer contextual information compared to vehicle-mounted cameras, leading to more accurate pedestrian behavior prediction.
  • Effective Feature Fusion: The proposed cross-stacked GRU architecture effectively fuses pedestrian posture, local context, and global context, demonstrating that combining these diverse information sources is crucial for robust prediction.
  • Posture as a Direct Indicator: Pedestrian posture (body and head) is identified as a direct and important indicator of intention, providing valuable input for the prediction model.
  • Impact of Pedestrian Speed: The study reveals that slower pedestrian speeds make prediction more difficult, particularly for negative samples (pedestrians not crossing). This is attributed to the ambiguity of slow-moving pedestrians who might be observing traffic or waiting for extended periods.
  • Time to Prediction (TTP) Sensitivity: As the prediction horizon (TTP) increases, the accuracy of all models tends to decrease, with the proposed model showing a more gradual decline, indicating better robustness.

Key Results

  • State-of-the-Art Performance: The proposed model achieved the best results across multiple metrics, including accuracy (0.80), AUC (0.80), F1 parameter (0.81), precision (0.75), and recall (0.88), outperforming baseline methods like Single RNN, Multi RNN, SF RNN, and PCPA.
  • Accuracy Above 80%: For a Time to Prediction (TTP) of 1.6 seconds, the model achieved an accuracy and F1 score exceeding 80%.
  • Qualitative Successes: The qualitative experiments showed instances where the proposed model correctly predicted crossing behavior when other models failed, particularly in cases where pedestrians exhibited subtle cues of intent (e.g., leg lifting while waiting).

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Field of View: Surveillance video, while better than vehicle cameras, still doesn't capture all relevant road information, especially on the right side of the camera's view. Pedestrians can perceive farther than the camera.
  • Complexity of Pedestrian-Vehicle Interaction: The rules governing pedestrian-vehicle interactions are complex and dynamic. The current dataset may not be sufficient to fully capture these nuances, leading to a decrease in model robustness with increasing TTP.
  • Reliance on Key Point Labeling: The current method relies on pre-annotated pedestrian key points, which limits its direct applicability in real-world scenarios where such annotations are not available.

Future work will focus on:

  • Multi-Camera Systems: Employing multiple cameras to broaden the observation field of view.
  • Dataset Augmentation: Increasing the size and diversity of the dataset to improve robustness and accuracy.
  • End-to-End Prediction: Developing a model that can directly detect human posture key points from surveillance video, enabling end-to-end prediction.
  • Alternative Posture Representations: Exploring different input representations for pedestrian posture beyond key points.

Summary updated

BibTeX
@misc{zhou2022prediction,
  author = {Xiao Zhou and Hongyu Ren and Tingting Zhang and Xingang Mou and Yi He and Ching-Yao Chan},
  title = {{Prediction of Pedestrian Crossing Behavior Based on Surveillance Video}},
  year = {2022},
  doi = {10.3390/s22041467},
}
Figure