PSI: A Pedestrian Behavior Dataset for Socially Intelligent Autonomous Car

Tina Chen, Renran Tian, Yaobin Chen, Joshua E. Domeyer, Heishiro Toyoda, Rini Sherony, Taotao Jing, Zhengming Ding

Introduction

Navigating urban environments remains a "wicked" problem for autonomous vehicles (AVs). While highway driving is largely governed by physical laws, city streets require social intelligence—the ability to interpret the goals, intents, and negotiations of pedestrians. Current trajectory prediction models often treat pedestrians as simple moving particles, failing to capture the underlying reasoning human drivers use to avoid accidents. To bridge this gap, researchers have introduced the PSI (Pedestrian Situated Intent) dataset and the eP2P (Explainable Pedestrian Trajectory Prediction) model, focusing on dynamic intent and human-like reasoning.

The PSI Dataset

The IUPUI-CSRC PSI dataset addresses two major gaps in existing benchmarks: the lack of dynamic intent labels and the absence of natural language explanations. Sampled from 1.4 million miles of naturalistic driving data, the dataset consists of 110 representative mid-block crossing cases.

Unlike traditional datasets that assign a single "cross" or "not cross" label to a video clip, PSI utilizes 24 different human annotators per video to provide frame-by-frame intent labels. This captures how a pedestrian's desire to cross shifts as a vehicle approaches. Furthermore, annotators provide text-based reasoning for their estimations, creating a rich multi-modal ground truth that includes semantic segmentation and visual pose estimation.

Overview of the PSI dataset annotations, including temporal intent graphs, visual frames, and semantic segmentation.

Methodology: The eP2P Model

The researchers developed the eP2P model, a multi-task learning framework designed to mimic human cognitive processes. The model architecture consists of several key components:

  1. Context Feature Extraction: A Convolutional-LSTM extracts features from the global scene and local pedestrian "crops," utilizing visual cues like pose and bounding boxes.
  2. Multi-Task Modules: Three parallel LSTM branches predict pedestrian intent (Cross, Not Cross, Not Sure), driver reasoning (text-based explanations), and an annotator disagreement score. The latter serves as a proxy for scene uncertainty.
  3. Trajectory Prediction: An integration module uses the outputs from the intent and reasoning branches to predict the future bounding box coordinates of the pedestrian.
Detailed visual annotations for a specific frame, showing the bounding boxes and skeleton-based pose estimation used for feature extraction.

Key Insights: Intent as a Fluid State

A core finding of the PSI project is that pedestrian intent is not a binary switch but a fluid state influenced by the vehicle's behavior. The dataset captures the "segmentation boundaries" where these shifts occur.

Furthermore, the "disagreement score" derived from the 24 annotators provides a valuable signal. High disagreement often occurs at critical decision points, allowing the AI to identify high-uncertainty situations. The inclusion of natural language reasoning also forces the model to learn more robust visual features, which directly translates to better trajectory accuracy.

Results and Conclusion

The eP2P model significantly outperforms established baselines like PIE and VR-GCN. In binary intent prediction, the model achieved an accuracy of 0.76 (vs. 0.69 for PIE). For trajectory prediction, it reduced the Average Displacement Error (ADE) for a 1.5-second horizon to 44.90, compared to 52.18 for the baseline.

While the dataset is currently limited in scale (110 scenes), its high annotation density provides a unique benchmark for socially intelligent AVs. Future work will focus on scaling the dataset and refining reasoning generation to support more complex social coordination between humans and autonomous systems.

Summary updated

BibTeX
@misc{chen2021psi,
  author = {Tina Chen and Renran Tian and Yaobin Chen and Joshua E. Domeyer and Heishiro Toyoda and Rini Sherony and Taotao Jing and Zhengming Ding},
  title = {{PSI: A Pedestrian Behavior Dataset for Socially Intelligent Autonomous Car}},
  year = {2021},
  eprint = {2112.02604},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.02604},
}
Figure