Pedestrian Behavior Prediction via Multitask Learning and Categorical Interaction Modeling

Amir Rasouli, Mohsen Rohani, Jun Luo

Abstract

Pedestrian behavior prediction is one of the major challenges for intelligent driving systems. Pedestrians often exhibit complex behaviors influenced by various contextual elements. To address this problem, we propose BiPed, a multitask learning framework that simultaneously predicts trajectories and actions of pedestrians by relying on multimodal data. Our method benefits from 1) a bifold encoding approach where different data modalities are processed independently allowing them to develop their own representations, and jointly to produce a representation for all modalities using shared parameters; 2) a novel interaction modeling technique that relies on categorical semantic parsing of the scenes to capture interactions between target pedestrians and their surroundings; and 3) a bifold prediction mechanism that uses both independent and shared decoding of multimodal representations. Using public pedestrian behavior benchmark datasets for driving, PIE and JAAD, we highlight the benefits of the proposed method for behavior prediction and show that our model achieves state-of-the-art performance and improves trajectory and action prediction by up to 22% and 9% respectively. We further investigate the contributions of the proposed reasoning techniques via extensive ablation studies.

Introduction

Predicting pedestrian intent is a critical component of autonomous driving. It requires more than just calculating velocity vectors; it necessitates an understanding of human intent within complex social and physical contexts. Most existing systems attempt to predict either a pedestrian's trajectory or their action (e.g., crossing the street). However, these tasks are intrinsically linked: a pedestrian's intent to cross influences their path, and their path reveals their intent. Predicting this from an ego-centric perspective—a camera mounted on a moving vehicle—is difficult because the camera's own motion (ego-motion) is confounded with the pedestrian's movement. Furthermore, pedestrians react to a variety of environmental factors, including vehicles, other pedestrians, and traffic infrastructure.

Methodology

The BiPed framework addresses these challenges through three key innovations: Bifold Encoding, Categorical Interaction Modeling, and Bifold Prediction. The architecture is designed to process multimodal data (trajectory, grid locations, and ego-vehicle motion) to produce consistent predictions across multiple tasks.

The BiPed architecture featuring Multimodal Independent Encoding (MIE), Multimodal Joint Encoding (MJE), the Categorical Interaction Module (CIM), and Bifold Prediction (MIP/MJP).

Bifold Encoding and Prediction

BiPed utilizes a "bifold" approach for both encoding and prediction. In the encoding phase, Multimodal Independent Encoding (MIE) allows each data stream to develop its own feature set, while Multimodal Joint Encoding (MJE) enables the model to learn correlations between modalities over time. Similarly, the Bifold Prediction mechanism employs both independent decoders (MIP) for specific tasks and a shared joint decoder (MJP). The final output is a fusion of these predictions, which acts as internal regularization to ensure logical consistency between the predicted path and action.

Categorical Interaction Module (CIM)

Rather than processing the entire scene as a single image, BiPed uses semantic segmentation to categorize the environment into specific groups: the target pedestrian, surrounding pedestrians, vehicles, and static objects (e.g., roads, sidewalks). The Categorical Interaction Module (CIM) then uses an Interaction Attention Unit (IAU) to weigh the importance of these categories relative to the target pedestrian's behavior.

Detailed view of the Categorical Interaction Module (CIM) which processes semantic maps into categorical representations to model scene interactions.

Experimental Results

The model was evaluated on two major benchmarks: PIE (Pedestrian Intention Estimation) and JAAD (Joint Attention in Autonomous Driving). BiPed demonstrated significant performance gains over previous state-of-the-art models:

  • Trajectory Prediction: On the PIE dataset, BiPed improved the Average Displacement Error (ADE) and Average RMSE of Bounding boxes (ARB) by up to 22%.
  • Action Prediction: The framework achieved a 9% improvement in F1-score and Precision for crossing intent prediction.
  • Ablation Studies: Results confirmed that the CIM alone improved trajectory metrics by up to 5% and bounding box accuracy by 12%, highlighting the importance of categorical context.

Conclusion

BiPed demonstrates that a multitask, bifold approach combined with categorical semantic reasoning significantly enhances the accuracy of pedestrian behavior prediction. While the model currently relies on high-quality semantic segmentation, its ability to handle noisy ego-motion data and coordinate path-action predictions sets a new standard for the field. Future work may explore applying this architecture to broader human-robot interaction tasks, such as gesture recognition or group activity prediction.

Summary updated

Abstract

Pedestrian behavior prediction is one of the major challenges for intelligent driving systems. Pedestrians often exhibit complex behaviors influenced by various contextual elements. To address this problem, we propose BiPed, a multitask learning framework that simultaneously predicts trajectories and actions of pedestrians by relying on multimodal data. Our method benefits from 1) a bifold encoding approach where different data modalities are processed independently allowing them to develop their own representations, and jointly to produce a representation for all modalities using shared parameters; 2) a novel interaction modeling technique that relies on categorical semantic parsing of the scenes to capture interactions between target pedestrians and their surroundings; and 3) a bifold prediction mechanism that uses both independent and shared decoding of multimodal representations. Using public pedestrian behavior benchmark datasets for driving, PIE and JAAD, we highlight the benefits of the proposed method for behavior prediction and show that our model achieves state-of-the-art performance and improves trajectory and action prediction by up to 22% and 9% respectively. We further investigate the contributions of the proposed reasoning techniques via extensive ablation studies.
BibTeX
@inproceedings{rasouli2020pedestrian,
  author = {Amir Rasouli and Mohsen Rohani and Jun Luo},
  title = {{Pedestrian Behavior Prediction via Multitask Learning and Categorical Interaction Modeling}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2020},
  doi = {10.48550/arxiv.2012.03298},
}

Loading related content...

Figure