Pedestrian Stop and Go Forecasting with Hybrid Feature Fusion

Dongxu Guo, Taylor Mordan, Alexandre Alahi

Introduction

Autonomous vehicles (AVs) excel at tracking pedestrians in steady motion, but the real danger lies in "abrupt dynamic changes"—the sudden decision to step into a street or halt at a curb. Traditional trajectory-based models, which rely on linear or smooth extrapolations of past coordinates, often fail to anticipate these critical transitions. This paper introduces the task of pedestrian stop and go forecasting and proposes a multi-modal solution to bridge this gap.

The TRANS Benchmark

To address the lack of specialized data for non-linear motion transitions, the authors present TRANS, a large-scale meta-dataset. By re-annotating and aggregating three major autonomous driving datasets—JAAD (crossing behaviors), PIE (roadside intentions), and TITAN (urban traffic in Tokyo)—the authors compiled a unified benchmark.

The resulting TRANS dataset contains over 2,100 transition events (1,008 "Go" and 1,138 "Stop") across nearly 9,000 unique pedestrians. This provides a diverse foundation for training models to recognize the subtle cues—social, behavioral, and environmental—that precede a change in state.

Hybrid Feature Fusion Architecture

The core contribution is a Hybrid Feature Fusion model designed to reason across four distinct modalities rather than relying on coordinates alone:

  1. Visual Features: Extracted via a ResNet-18 backbone. The model compares different context levels, finding that RoI-Context (global scene features via RoI-alignment) is the most effective for understanding the environment.
  2. Motion Features: Encoding the pedestrian’s 2D coordinates and velocity.
  3. Behavioral Attributes: Binary signals representing non-verbal communication, such as looking at traffic, nodding, or hand gestures.
  4. Scene Attributes: High-level semantic data about the environment, including lane counts, intersection types, and traffic flow direction.

Each modality is processed through its own Long Short-Term Memory (LSTM) network to capture temporal dependencies over a sequence of observations (typically 5–15 frames). These features are then fused and passed through a Multi-Layer Perceptron (MLP) to produce a binary prediction: transition vs. no-transition.

Key Insights and Performance

The research yields several critical insights into the nature of pedestrian behavior:

  • Context is King: Global scene context is significantly more effective than looking at the pedestrian in isolation. Knowing where the road, sidewalk, and crosswalk are provides the necessary "why" behind a pedestrian's decision to move.
  • Attribute Power: Adding high-level behavioral and scene attributes (like "looking" or "is an intersection") provides a massive performance boost—up to 23.2 points in Average Precision (AP) for "Go" forecasting on the JAAD dataset.
  • The Asymmetry of Motion: Predicting a "Go" transition (85.9% AP on JAAD) is generally easier than predicting a "Stop" (67.8% AP). This suggests that "Go" actions are often preceded by clear preparatory signals, whereas "Stops" can be more sudden or motivated by ambiguous factors.
  • Temporal Limits: Increasing the observation length improves performance up to approximately 10 frames (2 seconds). Beyond this point, adding more history can lead to stagnation as older frames become less relevant to the immediate future.

Limitations and Future Work

While the full hybrid model achieved state-of-the-art results, several challenges remain. The current model does not explicitly incorporate the AV's own speed or the state of traffic lights, both of which heavily influence pedestrian decisions. Furthermore, the experiments relied on ground-truth bounding boxes and attributes; in real-world deployments, these would be provided by noisy perception modules, potentially degrading performance. Future work will likely focus on more granular behavioral cues and the social interactions between multiple pedestrians.

Summary updated

Abstract

Forecasting pedestrians' future motions is essential for autonomous driving systems to safely navigate in urban areas. However, existing prediction algorithms often overly rely on past observed trajectories and tend to fail around abrupt dynamic changes, such as when pedestrians suddenly start or stop walking. We suggest that predicting these highly non-linear transitions should form a core component to improve the robustness of motion prediction algorithms. In this paper, we introduce the new task of pedestrian stop and go forecasting. Considering the lack of suitable existing datasets for it, we release TRANS, a benchmark for explicitly studying the stop and go behaviors of pedestrians in urban traffic. We build it from several existing datasets annotated with pedestrians' walking motions, in order to have various scenarios and behaviors. We also propose a novel hybrid model that leverages pedestrian-specific and scene features from several modalities, both video sequences and high-level attributes, and gradually fuses them to integrate multiple levels of context. We evaluate our model and several baselines on TRANS, and set a new benchmark for the community to work on pedestrian stop and go forecasting.
BibTeX
@inproceedings{guo2022pedestrian,
  author = {Dongxu Guo and Taylor Mordan and Alexandre Alahi},
  title = {{Pedestrian Stop and Go Forecasting with Hybrid Feature Fusion}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=tyf5OXPaybN},
}
Figure