Abstract
Pedestrian behavior prediction is one of the major challenges for intelligent driving systems. Pedestrians often exhibit complex behaviors influenced by various contextual elements. To address this problem, we propose BiPed, a multitask learning framework that simultaneously predicts trajectories and actions of pedestrians by relying on multi-modal data. Our method benefits from 1) a bifold encoding approach where different data modalities are processed independently and jointly; 2) a novel interaction modeling technique that relies on categorical semantic parsing; and 3) a bifold prediction mechanism that uses both independent and shared decoding. Using public pedestrian behavior benchmark datasets PIE and JAAD, we show that our model achieves state-of-the-art performance, improving trajectory and action prediction by up to 22% and 9% respectively.
Introduction
Predicting pedestrian behavior is a fundamental challenge for autonomous vehicles. Unlike vehicles, which are constrained by traffic lanes and laws, pedestrians move in a complex environment influenced by social interactions, environmental geometry, and the behavior of other road users. Accurate prediction requires understanding both the "where" (trajectory) and the "what" (action/intent).
Current systems often treat trajectory prediction and action prediction as separate tasks. However, these are deeply intertwined; a pedestrian's intent to cross directly influences their movement path. Furthermore, many models fail to capture the semantic context of interactions—distinguishing, for instance, between a pedestrian reacting to a moving bus versus a parked car. The BiPed framework aims to bridge this gap by integrating semantic context into a unified prediction loop.
The BiPed Framework
The BiPed framework is designed as a multitask learning system that processes multi-modal data, including trajectories, ego-motion, and grid locations, to predict future behavior.
Bifold Encoding and Prediction
BiPed utilizes a "bifold" architecture in both its encoding and prediction stages.
- Multimodal Independent Encoding (MIE): Processes each data type separately to preserve unique features and prevent noise in one modality from affecting others.
- Multimodal Joint Encoding (MJE): Processes modalities together to capture cross-modal correlations.
Similarly, the Multitask Independent Prediction (MIP) and Multitask Joint Prediction (MJP) modules use independent and shared decoders to output future trajectories, crossing probabilities, and final grid locations. Predicting the final grid location serves as a spatial anchor, helping the model reconcile continuous trajectory movement with binary action classification. This dual-path approach acts as a powerful regularizer, balancing specific modal features with global context.
Categorical Interaction Module (CIM)
A key innovation is the CIM, which provides semantic reasoning. It uses a pre-trained segmentation model to categorize the scene into five classes: target pedestrian, other pedestrians, vehicles, bicycles, and static context.
An Interaction Attention Unit (IAU) learns which categories are most relevant to the pedestrian's behavior, allowing the model to reason about why a pedestrian might be stopping or moving based on their surroundings rather than just raw 2D coordinates.
Experimental Evaluation
The model was evaluated on two benchmark datasets:
- PIE (Pedestrian Intention Estimation): Includes 6 hours of driving footage with rich vehicle sensor data.
- JAAD (Joint Attention in Autonomous Driving): Focused on driver-pedestrian interactions. Since JAAD lacks continuous ego-motion data, the authors compensated by utilizing high-level driver action labels.
For trajectory prediction, the authors employed the Log-cosh loss function, which offers greater robustness to outliers compared to standard L2 loss.
Results and Discussion
BiPed achieved significant improvements over previous state-of-the-art models, particularly on the PIE dataset:
- Trajectory Prediction: Improved Average Displacement Error (ADE) by 22% and bounding box accuracy (ARB) by 20% compared to the previous best model (PIEfull).
- Action Prediction: Achieved a 5-6% improvement in Accuracy and AUC, with Precision and F1 scores increasing by up to 9%.
- Qualitative Performance: The model demonstrated superior performance in complex scenes, correctly identifying "non-crossing" events where baseline models often produced false positives due to proximity to the curb.
Conclusion
The BiPed framework demonstrates that combining bifold data processing with semantic reasoning significantly enhances pedestrian behavior prediction. By treating trajectory and action prediction as a unified multitask problem and incorporating categorical environmental context, BiPed provides a more robust and "human-like" understanding of pedestrian intent, a critical step forward for safe autonomous navigation.
Summary updated