GPT-4V Takes the Wheel: Evaluating Promise and Challenges for Pedestrian Behavior Prediction

Jia Huang, Peng Jiang, Alvika Gautam, Srikanth Saripalli

Introduction

Predicting pedestrian behavior is a cornerstone of autonomous vehicle (AV) safety. While traditional deep learning models excel at pattern recognition within specific domains, they often lack the common-sense reasoning required to interpret complex human intentions. This paper investigates the zero-shot capabilities of GPT-4V(ision), a large Vision-Language Model (VLM), to determine if general-purpose AI can bridge the gap between raw data processing and causal reasoning in driving scenarios.

Methodology

The researchers evaluated GPT-4V using a zero-shot approach, ensuring the model relied on its pre-existing knowledge rather than domain-specific fine-tuning. The core of the methodology involved sophisticated prompt engineering and structured visual inputs.

Prompt structure for GPT-4V behavior prediction

To focus the model's attention, sequences of 10 frames were provided, with a red bounding box highlighting the pedestrian of interest. The system prompt cast GPT-4V as an AV assistant, with behavior definitions refined via ChatGPT to align with the model's internal logic. Experiments were conducted using the JAAD (Joint Attention in Autonomous Driving) dataset for quantitative benchmarking and the WiDEVIEW dataset for qualitative analysis.

Performance Analysis

The study revealed that while GPT-4V possesses impressive high-level reasoning, it currently lags behind state-of-the-art (SOTA) specialized models. GPT-4V achieved a 57% accuracy on the JAAD dataset, compared to approximately 70% for the SOTA model, PIT.

A critical finding was the "performance cliff" associated with pedestrian size. As shown in the performance metrics, accuracy and F-score significantly improve once a pedestrian occupies more than 1% of the total image area.

Performance metrics vs. pedestrian area ratio

Beyond size, the model struggled with "relative motion" and "subtle cues." GPT-4V frequently misinterpreted the motion of the ego-vehicle as the pedestrian's movement, leading to false "crossing" predictions. Furthermore, detecting "looking" behavior—which requires identifying head and eye orientation—remains a challenge due to current visual processing resolutions. Conversely, the model excelled at group logic, identifying collective intent more naturally than many specialized models.

Challenges and Future Outlook

Several significant "roadblocks" prevent the immediate deployment of VLMs in AV systems:

  • Latency: Processing times of 10–20 seconds per prompt are incompatible with the millisecond requirements of real-time driving.
  • Inconsistency: Even with deterministic settings (temperature 0), the model exhibited output variance across identical trials.
  • Visual Acuity: Higher resolution is necessary to capture the subtle facial expressions and head tilts vital for gauging pedestrian intent.

Future research should focus on behavior-specific fine-tuning and the development of efficient architectures to bring VLM reasoning to real-time edge devices.

Summary updated

Abstract

Predicting pedestrian behavior is the key to ensure safety and reliability of autonomous vehicles. While deep learning methods have been promising by learning from annotated video frame sequences, they often fail to fully grasp the dynamic interactions between pedestrians and traffic, crucial for accurate predictions. These models also lack nuanced common sense reasoning. Moreover, the manual annotation of datasets for these models is expensive and challenging to adapt to new situations. The advent of Vision Language Models (VLMs) introduces promising alternatives to these issues, thanks to their advanced visual and causal reasoning skills. To our knowledge, this research is the first to conduct both quantitative and qualitative evaluations of VLMs in the context of pedestrian behavior prediction for autonomous driving. We evaluate GPT-4V(ision) on publicly available pedestrian datasets: JAAD and WiDEVIEW. Our quantitative analysis focuses on GPT-4V's ability to predict pedestrian behavior in current and future frames. The model achieves a 57% accuracy in a zero-shot manner, which, while impressive, is still behind the state-of-the-art domain-specific models (70%) in predicting pedestrian crossing actions. Qualitatively, GPT-4V shows an impressive ability to process and interpret complex traffic scenarios, differentiate between various pedestrian behaviors, and detect and analyze groups. However, it faces challenges, such as difficulty in detecting smaller pedestrians and assessing the relative motion between pedestrians and the ego vehicle.
BibTeX
@inproceedings{huang2024gpt,
  author = {Jia Huang and Peng Jiang and Alvika Gautam and Srikanth Saripalli},
  title = {{GPT-4V Takes the Wheel: Evaluating Promise and Challenges for Pedestrian Behavior Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaaiss.v3i1.31192},
}
Figure