Coupling Intent and Action for Pedestrian Crossing Behavior Prediction

Yu Yao, Ella M. Atkins, Matthew Johnson-Roberson, Ram Vasudevan, Xiaoxiao Du

Introduction

For autonomous vehicles (AVs) to navigate complex urban environments safely, they must do more than just track objects; they must anticipate human behavior. Predicting whether a pedestrian is about to step into the road is a critical task. Traditional methods often rely on trajectory forecasting (X, Y coordinates) or pose estimation. However, these approaches frequently lack a "semantic" understanding of the pedestrian's underlying goals.

The fundamental challenge lies in the fact that intent—the internal decision to cross—is unobservable. A pedestrian standing at the curb might be waiting for a gap in traffic (intent to cross) or simply waiting for a bus (no intent to cross). This research proposes that intent and action exist in a feedback loop: a person's intent drives their current and future actions, and conversely, those future actions serve as the most reliable indicators of their underlying intent.

Proposed Framework: Coupling Intent and Action

The authors introduce a multi-task encoder-decoder network designed to explicitly couple these two elements. The framework consists of three primary modules:

  1. Intent-Action Encoder (ENC): This module processes visual features (via a VGG16 backbone) and spatial features (bounding box coordinates) to output the probability of binary intent (will cross vs. won't cross) alongside multi-class semantic actions.
  2. Future Action Predictor: This "look-ahead" component predicts a sequence of future semantic actions. Crucially, the features from these predicted future actions are fed back into the encoder as a prior, refining the detection of the current intent.
  3. Attentive Relation Network (ARN): To provide context, the ARN extracts features from surrounding traffic objects, such as neighboring vehicles, traffic lights, and crosswalks.
Figure 1: Sequence of pedestrian actions and intent over time, showing how future actions inform current intent classification.

By training intent and action detection jointly, the model learns that specific actions (e.g., "waiting" at a curb) are highly correlated with specific intents, leading to more explainable and robust predictions.

Environmental Context and Relation Modeling

A key innovation is the Attentive Relation Network (ARN). Unlike previous methods that relied on computationally expensive dense semantic maps, the ARN focuses on discrete, pre-detected traffic objects. This mirrors human driving logic—drivers look for specific cues like crosswalks and signal lights rather than processing every pixel in a scene.

The ARN utilizes a Soft Attention mechanism to weigh the relevance of environmental factors. For instance, a red traffic light is significantly more relevant to a waiting pedestrian than a distant parked car. This selective focus allows the model to understand the "why" behind a pedestrian's behavior.

Datasets and Evaluation

The model was evaluated on two major naturalistic driving benchmarks:

  • PIE (Pedestrian Intent Estimation): A large-scale dataset with 1,842 pedestrians.
  • JAAD (Joint Attention in Autonomous Driving): Contains 346 videos with 686 pedestrians.

To improve granularity, the authors expanded the original binary labels into seven semantic categories: standing, waiting, going towards, crossing, crossed and standing, crossed and walking, and other walking.

Key Results

The proposed multi-task framework demonstrates significant improvements over existing methods:

  • State-of-the-Art Accuracy: On the PIE dataset, the model achieved an AUC of 0.83 and a $\Delta_s$ of 0.41, significantly outperforming the previous state-of-the-art PIEint model (0.73 AUC).
  • Environmental Impact: Ablation studies confirmed that traffic lights and crosswalks are the most critical features for accurate prediction, validating the model's reliance on human-intuitive cues.
  • Real-Time Efficiency: The entire inference pipeline runs in approximately 6ms on an NVIDIA Tesla V100 GPU, making it suitable for real-time AV deployment.

Limitations and Future Work

While effective, the model's performance is tied to the accuracy of upstream object detectors. If a crosswalk or traffic light is missed by the detector, the ARN's effectiveness drops. Additionally, the model occasionally struggles to balance a pedestrian's own motion against environmental context (e.g., distinguishing between waiting for a bus vs. waiting to cross). Future research may explore semi-supervised methods to incorporate "unseen" or unexpected environmental objects into the training process.

Summary updated

Abstract

Accurate prediction of pedestrian crossing behaviors by autonomous vehicles can significantly improve traffic safety. Existing approaches often model pedestrian behaviors using trajectories or poses but do not offer a deeper semantic interpretation of a person's actions or how actions influence a pedestrian's intention to cross in the future. In this work, we follow the neuroscience and psychological literature to define pedestrian crossing behavior as a combination of an unobserved inner will (a probabilistic representation of binary intent of crossing vs. not crossing) and a set of multi-class actions (e.g., walking, standing, etc.). Intent generates actions, and the future actions in turn reflect the intent. We present a novel multi-task network that predicts future pedestrian actions and uses predicted future action as a prior to detect the present intent and action of the pedestrian. We also designed an attention relation network to incorporate external environmental contexts thus further improve intent and action detection performance. We evaluated our approach on two naturalistic driving datasets, PIE and JAAD, and extensive experiments show significantly improved and more explainable results for both intent detection and action prediction over state-of-the-art approaches. Our code is available at: https://github.com/umautobots/pedestrian_intent_action_detection.
BibTeX
@inproceedings{yao2021coupling,
  author = {Yu Yao and Ella M. Atkins and Matthew Johnson-Roberson and Ram Vasudevan and Xiaoxiao Du},
  title = {{Coupling Intent and Action for Pedestrian Crossing Behavior Prediction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/171},
}
Figure