Multi-Task Deep Learning for Pedestrian Detection, Action Recognition and Time to Cross Prediction

Danut Ovidiu Pop, Alexandrina Rogozan, Clément Chatelain, Fawzi Nashashibi, Abdelaziz BensrhairIEEE Access 2019 · OpenReview

Abstract

A pedestrian detection system is a crucial component of advanced driver assistance systems since it contributes to road flow safety. The safety of traffic participants could be significantly improved if these systems could also predict and recognize pedestrian's actions, or even estimate the time, for each pedestrian, to cross the street. In this paper, we focus not only on pedestrian detection and pedestrian action recognition but also on estimating if the pedestrian's action presents a risky situation according to time to cross the street. We propose 1) a pedestrian detection and action recognition component based, on RetinaNet; 2) an estimation of the time to cross the street for multiple pedestrians using a recurrent neural network. For each pedestrian, the recurrent network estimates the pedestrian's action intention in order to predict the time to cross the street. We based our experiments on the JAAD dataset, and show that integrating multiple pedestrian action tags for the detection part when merge with a recurrent neural network (LSTM) allows a significant performance improvement.

Introduction

Modern Advanced Driver Assistance Systems (ADAS) have become highly proficient at detecting pedestrians. However, detection alone is often insufficient for ensuring safety in complex urban environments. A critical missing piece is the "intuition" to understand human intent—distinguishing between a person waiting for a bus and one about to step into traffic.

This research addresses three primary gaps: the ambiguity of intent in current systems, the lack of robust temporal prediction models for "Time to Cross" (TTC), and the scarcity of datasets with explicit TTC annotations. By integrating detection, action recognition, and temporal prediction, the proposed framework aims to provide vehicles with a more comprehensive understanding of pedestrian behavior.

Methodology

The proposed system utilizes a two-stage pipeline designed for multi-task learning.

Detection and Action Recognition

The first stage employs a RetinaNet architecture with a ResNet50 backbone. Unlike standard detectors that only identify "pedestrian" vs. "background," this model's classification head is modified to recognize four distinct behavioral states:

  • PPC (Preparing to cross): The pedestrian is stationary but showing signs of intent to cross.
  • PC (Crossing): The pedestrian is currently in the roadway.
  • PAC (About to cross): The pedestrian is at the edge of the curb, ready to step off.
  • PA (Ambiguous): The intention cannot be clearly determined from visual cues.

Time to Cross (TTC) Prediction

The second stage uses a Long Short-Term Memory (LSTM) network to process temporal sequences. The LSTM receives a sequence of bounding box coordinates and semantic action tags from previous frames. By analyzing history (e.g., up to 40 frames), the model predicts the number of frames remaining until the pedestrian initiates a crossing action.

Proposed multi-task architecture combining CNN-based feature extraction with an LSTM for temporal prediction.

Data and Training

The researchers utilized the JAAD (Joint Attention for Autonomous Driving) dataset. Because JAAD lacks native TTC values, the authors manually annotated trajectories to create ground truth labels for regression-based training.

Results and Discussion

The integration of semantic action labels proved to be a significant factor in prediction accuracy.

Performance Metrics

The general pedestrian detector achieved a 56.05% mAP. When tasked with fine-grained action classification, the mAP was 26.36%, reflecting the difficulty of distinguishing subtle behavioral cues. Despite this, the TTC prediction remained robust. For pedestrians in the "Preparing to Cross" (PPC) state, the model achieved a Root Mean Square Error (RMSE) of 4.76% on real values.

Pedestrian detection and action recognition in a real-world urban environment.

Key Insights

  • Semantics Matter: Incorporating action tags like "Preparing to Cross" into the LSTM input significantly outperformed models using only raw bounding box coordinates.
  • Temporal Context: Accuracy improved as the temporal window increased from 5 to 40 frames, highlighting the importance of long-term history in predicting human intent.
  • Robustness: The system maintained low error rates even when using predicted (imperfect) bounding boxes, suggesting it is viable for real-world deployment.

Conclusion

This work demonstrates that multi-task learning, combining spatial detection with temporal action recognition, significantly enhances a vehicle's ability to predict pedestrian crossing times. Future work will focus on incorporating environmental context—such as crosswalk locations and traffic signals—and developing end-to-end models to reduce computational complexity for real-time applications.

Summary updated

Abstract

A pedestrian detection system is a crucial component of advanced driver assistance systems since it contributes to road flow safety. The safety of traffic participants could be significantly improved if these systems could also predict and recognize pedestrian's actions, or even estimate the time, for each pedestrian, to cross the street. In this paper, we focus not only on pedestrian detection and pedestrian action recognition but also on estimating if the pedestrian's action presents a risky situation according to time to cross the street. We propose 1) a pedestrian detection and action recognition component based, on RetinaNet; 2) an estimation of the time to cross the street for multiple pedestrians using a recurrent neural network. For each pedestrian, the recurrent network estimates the pedestrian's action intention in order to predict the time to cross the street. We based our experiments on the JAAD dataset, and show that integrating multiple pedestrian action tags for the detection part when merge with a recurrent neural network (LSTM) allows a significant performance improvement.
BibTeX
@inproceedings{pop2019multi,
  author = {Danut Ovidiu Pop and Alexandrina Rogozan and Clément Chatelain and Fawzi Nashashibi and Abdelaziz Bensrhair},
  title = {{Multi-Task Deep Learning for Pedestrian Detection, Action Recognition and Time to Cross Prediction}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=GvcjwpgVXM},
}
Figure