Learning To Detect Phone-Related Pedestrian Distracted Behaviors With Synthetic Data

Emre Hatay, Jin Ma, Huiming Sun, Jianwu Fang, Zhiqiang Gao, Hongkai YuWorkshops @ CVPR 2021 · DOI

Introduction

Pedestrian fatalities are on the rise, and smartphone distraction—texting, gaming, or calling while crossing the street—is a major culprit. While autonomous vehicles and advanced driver-assistance systems (ADAS) use computer vision to spot pedestrians, specifically identifying distracted behavior remains a significant challenge. The difficulty stems from two primary issues: the scarcity of labeled real-world data for dangerous behaviors and the need for temporal context to distinguish distraction from normal movement.

Methodology: PPDBNet

To address these challenges, researchers developed PPDBNet, a deep learning model designed to improve the detection of phone-distracted pedestrians from vehicle-mounted cameras.

Siamese-LSTM Architecture

The model processes synchronized video feeds from two cameras (left and right) installed on a vehicle. It utilizes a VGG16 backbone to extract spatial features and Long Short-Term Memory (LSTM) layers to capture the temporal "flow" of behavior over time. This approach recognizes that distraction is a process rather than a single-frame event.

The PPDBNet architecture featuring dual-branch spatial-temporal processing of image and pose data.

Pose-Aware Feature Fusion and the "Pose Bridge"

A key innovation is the incorporation of 2D human pose estimation (keypoints of the skeleton). By focusing on posture rather than raw pixels, the model becomes more robust to background clutter. Furthermore, the researchers used 2D poses as a domain-invariant feature to bridge the "domain gap" between synthetic and real-world data. Because a skeleton looks mathematically similar in a game engine and real life, it serves as a reliable anchor for transfer learning.

Datasets

The study utilizes two primary datasets to train and validate the system:

  • SYN-PPDB (Synthetic): Created using the Unity 3D engine, containing 448 synchronized video pairs featuring 10 virtual actors in various city environments.
  • PPDB (Real-world): A benchmark dataset of 448 synchronized video pairs collected from a real vehicle, used for fine-tuning and evaluation.

Behaviors are categorized into three classes: No Engagement, Eye Engagement (texting/gaming), and Phone Call Engagement.

Key Results

The proposed PPDBNet-S2R-FT model achieved a state-of-the-art 96.67% overall accuracy, significantly outperforming previous methods such as Dual-branch CNNs and MS-G3D.

Performance comparison of PPDBNet against state-of-the-art methods across three distraction classes.

Notably, the model achieved 100% accuracy for "Class 2" (Eye Engagement), the most safety-critical behavior. The system is also efficient, processing a 10-frame sub-sequence in approximately 108ms, making it viable for near real-time deployment in smart vehicles.

Conclusion and Future Work

While PPDBNet sets a new benchmark, limitations remain. There is an inherent trade-off between sub-sequence length (accuracy) and processing speed. Additionally, some "ambiguous" actions, such as using a speakerphone, can still lead to minor misclassifications. Future research will focus on increasing environmental variety in synthetic data—including diverse weather and lighting—to further harden the model for complex urban edge cases.

Summary updated

BibTeX
@inproceedings{hatay2021learning,
  author = {Emre Hatay and Jin Ma and Huiming Sun and Jianwu Fang and Zhiqiang Gao and Hongkai Yu},
  title = {{Learning To Detect Phone-Related Pedestrian Distracted Behaviors With Synthetic Data}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  doi = {10.1109/cvprw53098.2021.00333},
}
Figure