Gating Syn-to-Real Knowledge for Pedestrian Crossing Prediction in Safe Driving

Jie Bai, Jianwu Fang, Yisheng Lv, Chen Lv, Jianru Xue, Zhengguo Li

TL;DR

Predicting whether a pedestrian will cross the street is vital for autonomous vehicle safety, but real-world training data for these rare or dangerous scenarios is hard to come by. This paper introduces Gated-S2R-PCP, a framework that leverages rich synthetic data by applying tailored domain-adaptation techniques to different data modalities based on their specific "domain gaps." By adaptively fusing these specialized insights using a learnable gating unit, the model achieves state-of-the-art pedestrian crossing prediction on real-world datasets.

What Problem Does This Paper Address?

Pedestrian Crossing Prediction (PCP) is a challenging task for intelligent vehicles. To safely navigate urban environments, a vehicle must look at a pedestrian for a split second (e.g., 0.5 seconds) and predict their crossing intention seconds into the future.

While deep learning has advanced PCP, models suffer from a lack of diverse real-world training data—especially for dangerous near-collision scenarios. While researchers often turn to synthetic data (like simulations from CARLA) to fill the gap, transferring knowledge from simulation to reality ("Syn-to-Real") introduces a major hurdle: the domain gap.

The authors observe that a "one-size-fits-all" domain adaptation approach fails because different types of information have vastly different domain gaps. As shown via t-SNE analysis in the paper, synthetic pedestrian trajectories (bounding boxes) map very closely to real ones. However, synthetic RGB images have a moderate gap due to style differences, and synthetic depth or semantic maps have massive gaps when compared to real-world raw video frames.

Differentiated domain gaps between synthetic and real data modalities

Methods

To address these uneven domain gaps, the authors propose Gated-S2R-PCP, a framework that divides domain adaptation into three specialized pathways, tied together by an intelligent gating mechanism:

Overview of the Gated-S2R-PCP framework
  1. Knowledge Distiller (KnowD): For pedestrian location data (bounding boxes), which has a small domain gap. A teacher model trained on synthetic data guides a lightweight student model (ResNet+LSTM) processing real data. Crucially, it includes an auxiliary task: predicting the Future Object Location (FOL), which forces the model to learn motion consistency.
  2. Style Shifter (StyS): For RGB video frames, which have a moderate domain gap. It uses Adaptive Instance Normalization (AdaIN) to perform feature-level style transfer, altering synthetic images to match real-world environmental and lighting conditions.
  3. Distribution Approximator (DistA): For semantic and depth images, which have large domain gaps. It uses a Generative Adversarial Network (GAN) style setup with a Gradient Reversal Layer (GRL) to project these abstract synthetic modalities and real RGB frames into a shared, domain-invariant feature space.

Learnable Gated Unit (LGU)

Instead of statically merging these features, the framework passes them through a Learnable Gated Unit (LGU). The LGU dynamically weights the importance of location, visual style, and semantic/depth clues depending on the specific driving scenario before making the final crossing prediction.

Structure of the Learnable Gated Unit

Datasets

The authors evaluated their approach using both synthetic and real-world benchmarks:

  • Syn-PCP-3181 (New): A large-scale synthetic dataset created by the authors using the CARLA simulator. It contains 3,181 video sequences (489,740 frames) complete with RGB, depth, semantic segmentation, and pedestrian bounding boxes across balanced weather and lighting conditions.
  • JAAD & PIE: Two widely used real-world datasets captured in actual driving environments, used here to test the model's real-world generalization.
  • PC50-DADA: A specialized 50-sequence subset of the DADA-2000 crash dataset, used to evaluate how well the model predicts sudden, near-collision pedestrian crossings.

Key Insights and Ideas

  • Differentiated Adaptation is Key: Treating bounding boxes, raw pixels, and depth maps with the same domain-transfer technique causes "domain confusion." Tailoring the tool to the size of the gap yields cleaner feature representations.
  • Intent Follows Motion: Incorporating Future Object Location (FOL) prediction as an auxiliary task significantly boosts PCP accuracy. Teaching the model where a pedestrian will physically be helps it better understand what the pedestrian intends to do.
  • Naïve Mixing Fails: The authors showed that simply combining synthetic and real data into one big training bucket ("Syn+Real") actually degrades performance compared to training on real data alone. Structured, gated knowledge transfer is mandatory.

Key Results

Gated-S2R-PCP demonstrated superior performance over existing state-of-the-art (SOTA) methods:

  • On the PIE Dataset: Achieved an accuracy of 0.92 and an AUC of 0.90, outperforming standard baselines like PCPA and matching or beating highly complex transformer architectures.
  • Ablation Success: Fusing all three modules (KnowD + StyS + DistA) via the LGU provided the highest overall accuracy, proving that the gating mechanism successfully selects the right features at the right time.
  • Near-Collision Performance: On the brutal PC50-DADA dataset—where pedestrians suddenly step into traffic—the model achieved an overall accuracy of 0.66. While lower than standard datasets, it successfully demonstrated that prediction accuracy increases as the pedestrian gets closer to the vehicle (rising to 0.63 at a 1.0-second Time-to-Crossing window).

Limitations and Future Work

Despite the framework's success, the authors note that near-collision prediction remains an open challenge. In scenarios where pedestrians appear suddenly from behind obstacles, the model's accuracy drops, indicating a massive space for improvement in rapid-reaction perception.

Additionally, while auxiliary FOL prediction helps short-term accuracy, attempting to predict pedestrian locations too far into the future (e.g., 1.0 second or more) introduces erratic behavioral noise that can actually hurt the model's recall. Future work will likely focus on tightening these prediction windows and improving robustness in occluded environments.

Summary updated

Abstract

Pedestrian Crossing Prediction (PCP) in driving scenes plays a critical role in ensuring the safe operation of intelligent vehicles. Due to the limited observations of pedestrian crossing behaviors in typical situations, recent studies have begun to leverage synthetic data with flexible variation to boost prediction performance, employing domain adaptation frameworks. However, different domain knowledge has distinct cross-domain distribution gaps, which necessitates suitable domain knowledge adaption ways for PCP tasks. In this work, we propose a Gated Syn-to-Real Knowledge transfer approach for PCP (Gated-S2R-PCP), which has two aims: 1) designing the suitable domain adaptation ways for different kinds of crossing-domain knowledge, and 2) transferring suitable knowledge for specific situations with gated knowledge fusion. Specifically, we design a framework that contains three domain adaption methods including style transfer, distribution approximation, and knowledge distillation for various information, such as visual, semantic, depth, location, etc. A Learnable Gated Unit (LGU) is employed to fuse suitable cross-domain knowledge to boost pedestrian crossing prediction. We construct a new synthetic benchmark S2R-PCP-3181 with 3181 sequences (489,740 frames) which contains the pedestrian locations, RGB frames, semantic images, and depth images. With the synthetic S2R-PCP-3181, we transfer the knowledge to two real challenging datasets of PIE and JAAD, and superior PCP performance is obtained to the state-of-the-art methods.
BibTeX
@article{bai2025gating,
  author = {Jie Bai and Jianwu Fang and Yisheng Lv and Chen Lv and Jianru Xue and Zhengguo Li},
  title = {{Gating Syn-to-Real Knowledge for Pedestrian Crossing Prediction in Safe Driving}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2025},
  doi = {10.1109/tits.2025.3554767},
}
Figure