TL;DR
This paper argues that traditional metrics for pedestrian behavior prediction are insufficient for automated driving (AD) systems. They introduce a new system-level metric, In-ROI Sensitivity (IRS), which directly evaluates how well predictions support AD system reactions, and demonstrate its effectiveness through an ablation study on a novel, large-scale dataset.
What Problem Does This Paper Address?
Automated vehicles (AVs) need to predict pedestrian behavior to drive safely and comfortably. However, existing methods for pedestrian prediction often use generic evaluation metrics that don't fully capture the needs of downstream AD functions. This can lead to models that are overly complex or suboptimal for the actual task. The authors highlight that predicting pedestrian behavior isn't just about accuracy, but about how well those predictions enable the AV to react appropriately, especially in complex, long-term scenarios. They aim to bridge this gap by defining system-level requirements and developing a metric that reflects these needs.
Methods
The core of the paper's contribution lies in its system-level approach. First, they analyze human driving behavior in pedestrian interaction scenarios to derive "comfort zones" and appropriate AD system reaction patterns. This analysis leads to the definition of requirements for pedestrian prediction models, emphasizing the need for probabilistic, multi-modal predictions over longer horizons.
Based on these requirements, they propose a novel evaluation metric called In-ROI Sensitivity (IRS). This metric treats pedestrian prediction as a binary classification problem: predicting whether a pedestrian will enter a defined "Region of Interest" (ROI) – essentially the AV's future driving corridor. IRS measures the True Positive Rate (TPR) at specific False Positive Rate (FPR) thresholds, tailored to different prediction horizons. The rationale is that for longer prediction horizons, a higher false positive rate is acceptable as it allows for more flexibility in planning, whereas short horizons demand higher precision to avoid critical situations.
For their prediction model, they employ a Conditional Variational Autoencoder (CVAE). This choice is motivated by its ability to model complex, multi-modal probability distributions and its explicit density modeling for NLL evaluation. The CVAE is designed to incorporate various contextual cues, including past pedestrian motion, semantic map information of the environment, and ego-vehicle dynamics.
Datasets
The authors introduce a new, large-scale dataset specifically designed for pedestrian behavior prediction in an AD context. This dataset comprises:
- Vehicle-pedestrian interactions recorded in southern Germany.
- Data from three different round-courses to ensure scenario variability.
- 3D point clouds from LiDAR, images from front-facing cameras, and IMU/GPS data for ego-vehicle localization.
- Crucially, it includes hand-labeled semantic maps of the road infrastructure and detailed pedestrian attributes (body/head orientation).
They collected 74 hours of data, resulting in over 93,000 unique pedestrian tracks. A subset of 9,438 tracks was manually labeled for features relevant to their study. They also compare their findings to existing datasets like STIP, JAAD, PIE, TITAN, SDD, INTERACTION, inD, Argoverse, and nuScenes, noting their limitations in terms of representation or available features.
Key Insights and Ideas
- System-level evaluation is crucial: Generic metrics like ADE and NLL can be misleading. A metric that directly reflects the downstream AD system's needs (like IRS) is more informative for model selection and development.
- Long-term prediction demands context: While simple motion models suffice for very short prediction horizons (e.g., 1s), incorporating contextual cues (map, egodist, head/body pose) becomes increasingly important for longer horizons (2s-4s) to achieve good performance.
- Not all features are equally valuable for all horizons: The importance of features shifts with the prediction horizon. Egocentric distance and head/body pose are beneficial for short-term predictions, while semantic map information is critical for longer-term predictions.
- Human driving behavior provides a blueprint: Analyzing how human drivers interact with pedestrians offers valuable insights into desired AV behavior and helps define appropriate prediction requirements. The concept of a "comfort zone" for time-to-collision (TTC) is a key takeaway.
- Trade-offs in prediction: The paper highlights that models optimized for traditional metrics (like NLL) might not be optimal for system-level performance. For instance, a model that accurately predicts a multi-modal distribution might have a higher NLL than a simpler, uni-modal model, but perform better according to IRS.
Key Results
- The proposed IRS metric effectively differentiates prediction models based on their utility for AD systems, showing that models incorporating contextual cues significantly outperform simpler motion-only models for longer prediction horizons (2s-4s).
- An ablation study revealed that the semantic map is a critical feature for improving prediction performance, especially for longer horizons.
- For a 1s prediction horizon, a simple CVAE motion model achieved high IRS scores (around 96%), indicating that contextual cues offer only minor benefits. However, for a 4s prediction horizon, the full feature set CVAE achieved an IRS of 93.3%, a substantial improvement over the motion-only model (82.6%).
- The authors found that traditional metrics like NLL and ADE sometimes suggested different optimal feature sets compared to IRS. For example, NLL and ADE favored models with all features, while IRS indicated that "motion + map" was often sufficient.
- Qualitative examples demonstrated how map features help the model predict more nuanced behaviors, such as a pedestrian's increased likelihood of crossing at a zebra crossing or reacting to a decelerating vehicle.
Limitations and Future Work
- Dataset bias: The dataset was collected in southern Germany, which might limit the generalizability of trained models to regions with different cultural factors or traffic rules.
- Feature relevance variability: While the study identified key features, the authors note that adding egodist or head&body features did not always yield significant improvements for 3s and 4s predictions, possibly due to limitations in the available labeled data for these features.
- Future work suggestions:
- Extend the ablation study to other datasets and consider more contextual cues, such as interactions with other traffic participants (multiple agents).
- Evaluate additional model types, particularly multi-agent prediction models.
- Focus on integrating and optimizing the developed prediction component within a full AD system.
Summary updated