Coupling Intent and Action for Pedestrian Crossing Behavior Prediction

Yu Yao, Ella M. Atkins, Matthew Johnson-Roberson, Ram Vasudevan, Xiaoxiao Du

Abstract

Accurate prediction of pedestrian crossing behaviors by autonomous vehicles can significantly improve traffic safety. Existing approaches often model pedestrian behaviors using trajectories or poses but do not offer a deeper semantic interpretation of a person's actions or how actions influence a pedestrian's intention to cross in the future. In this work, we follow the neuroscience and psychological literature to define pedestrian crossing behavior as a combination of an unobserved inner will (a probabilistic representation of binary intent of crossing vs. not crossing) and a set of multi-class actions (e.g., walking, standing, etc.). Intent generates actions, and the future actions in turn reflect the intent. We present a novel multi-task network that predicts future pedestrian actions and uses predicted future action as a prior to detect the present intent and action of the pedestrian. We also designed an attention relation network to incorporate external environmental contexts thus further improve intent and action detection performance. We evaluated our approach on two naturalistic driving datasets, PIE and JAAD, and extensive experiments show significantly improved and more explainable results for both intent detection and action prediction over state-of-the-art approaches.

1. Introduction

For an autonomous vehicle (AV) to navigate safely, it doesn't just need to know where a pedestrian is; it needs to know what they intend to do. Most existing models treat pedestrian behavior as a trajectory prediction problem—essentially guessing the next set of coordinates. However, this lacks semantic depth and fails to capture the underlying psychological drivers of movement.

The authors argue that pedestrian behavior is a feedback loop: a person’s hidden intent (e.g., "I want to cross") drives their actions (e.g., "walking toward the curb"), and those actions, in turn, confirm the intent. Furthermore, pedestrians do not exist in a vacuum; they react to traffic lights, crosswalks, and other vehicles. The challenge is building a model that captures this psychological "will," the physical "action," and the environmental "context" simultaneously.

Pedestrian crossing behavior as a sequence of actions driven by intent.

2. Methodology

The researchers proposed a novel multi-task network consisting of three main components designed to mimic human cognitive processes:

  1. Intent-Action Encoder (ENC): This module processes visual features (from a CNN) and location features (bounding box coordinates) to estimate the pedestrian's current intent (binary: will cross/won't cross) and current action (multi-class: standing, walking, etc.).
  2. Future Action Predictor (DEC): An encoder-decoder structure that forecasts a sequence of future actions. Crucially, these predicted future actions are looped back into the encoder as a "prior" for the next time step. This mimics human cognition: if we see someone speeding up toward a crosswalk, we use that "future" expectation to confirm our "present" belief that they intend to cross.
  3. Attentive Relation Network (ARN): Instead of using raw semantic maps, the ARN extracts specific features from six types of traffic objects: neighbors (other cars/pedestrians), traffic lights, signs, crosswalks, stations, and the ego-car’s own motion. It uses a "soft attention" mechanism to decide which of these objects are most relevant to the pedestrian's decision at any given moment.
The proposed multi-task network architecture for intent and action prediction.

3. Experimental Setup

The model was evaluated on two major naturalistic driving benchmarks:

  • PIE (Pedestrian Intent Estimation): A large-scale dataset with over 1,800 pedestrians and behavioral tags.
  • JAAD (Joint Attention in Autonomous Driving): Focused on crossing intent at intersections.

The authors notably augmented the original binary action labels (standing/walking) into seven semantic categories (e.g., "going towards," "crossing," "crossed and walking") to provide more granular training data for their multi-task framework.

4. Key Insights and Results

The core innovation is treating action as a mirror of intent. By forcing the model to predict the future and then use that prediction to justify the present, the system becomes much more robust. Furthermore, reasoning about specific objects (e.g., distance to a stop sign) proved more effective than processing global semantic maps, as it is more computationally efficient and aligns better with how human drivers scan a scene.

The full model outperformed previous state-of-the-art models (like PIEint and SF-GRU) by significant margins, particularly in the $\Delta_s$ metric, which measures the clarity of the model's distinction between "crossing" and "not crossing." Ablation studies showed that traffic lights and crosswalks are the most critical environmental features. Despite its complexity, the system remains efficient, with an inference time of roughly 6ms per frame on a standard GPU.

5. Conclusion and Future Work

While the model shows significant promise, it remains dependent on visible infrastructure. In scenes without clear crosswalks or traffic lights, confidence can drop. Future work will focus on reducing dependency on pre-detected objects and exploring unsupervised methods to incorporate novel environmental contexts that the model was not specifically trained to recognize.

Summary updated

Abstract

Accurate prediction of pedestrian crossing behaviors by autonomous vehicles can significantly improve traffic safety. Existing approaches often model pedestrian behaviors using trajectories or poses but do not offer a deeper semantic interpretation of a person's actions or how actions influence a pedestrian's intention to cross in the future. In this work, we follow the neuroscience and psychological literature to define pedestrian crossing behavior as a combination of an unobserved inner will (a probabilistic representation of binary intent of crossing vs. not crossing) and a set of multi-class actions (e.g., walking, standing, etc.). Intent generates actions, and the future actions in turn reflect the intent. We present a novel multi-task network that predicts future pedestrian actions and uses predicted future action as a prior to detect the present intent and action of the pedestrian. We also designed an attention relation network to incorporate external environmental contexts thus further improve intent and action detection performance. We evaluated our approach on two naturalistic driving datasets, PIE and JAAD, and extensive experiments show significantly improved and more explainable results for both intent detection and action prediction over state-of-the-art approaches. Our code is available at: https://github.com/umautobots/pedestrian_intent_action_detection.
BibTeX
@inproceedings{yao2021coupling,
  author = {Yu Yao and Ella M. Atkins and Matthew Johnson-Roberson and Ram Vasudevan and Xiaoxiao Du},
  title = {{Coupling Intent and Action for Pedestrian Crossing Behavior Prediction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/171},
}
Figure