TL;DR
Predicting whether a pedestrian will cross the street is vital for autonomous vehicle safety, but real-world training data for these rare or dangerous scenarios is hard to come by. This paper introduces Gated-S2R-PCP, a framework that leverages rich synthetic data by applying tailored domain-adaptation techniques to different data modalities based on their specific "domain gaps." By adaptively fusing these specialized insights using a learnable gating unit, the model achieves state-of-the-art pedestrian crossing prediction on real-world datasets.
What Problem Does This Paper Address?
Pedestrian Crossing Prediction (PCP) is a challenging task for intelligent vehicles. To safely navigate urban environments, a vehicle must look at a pedestrian for a split second (e.g., 0.5 seconds) and predict their crossing intention seconds into the future.
While deep learning has advanced PCP, models suffer from a lack of diverse real-world training data—especially for dangerous near-collision scenarios. While researchers often turn to synthetic data (like simulations from CARLA) to fill the gap, transferring knowledge from simulation to reality ("Syn-to-Real") introduces a major hurdle: the domain gap.
The authors observe that a "one-size-fits-all" domain adaptation approach fails because different types of information have vastly different domain gaps. As shown via t-SNE analysis in the paper, synthetic pedestrian trajectories (bounding boxes) map very closely to real ones. However, synthetic RGB images have a moderate gap due to style differences, and synthetic depth or semantic maps have massive gaps when compared to real-world raw video frames.
Methods
To address these uneven domain gaps, the authors propose Gated-S2R-PCP, a framework that divides domain adaptation into three specialized pathways, tied together by an intelligent gating mechanism:
- Knowledge Distiller (KnowD): For pedestrian location data (bounding boxes), which has a small domain gap. A teacher model trained on synthetic data guides a lightweight student model (ResNet+LSTM) processing real data. Crucially, it includes an auxiliary task: predicting the Future Object Location (FOL), which forces the model to learn motion consistency.
- Style Shifter (StyS): For RGB video frames, which have a moderate domain gap. It uses Adaptive Instance Normalization (AdaIN) to perform feature-level style transfer, altering synthetic images to match real-world environmental and lighting conditions.
- Distribution Approximator (DistA): For semantic and depth images, which have large domain gaps. It uses a Generative Adversarial Network (GAN) style setup with a Gradient Reversal Layer (GRL) to project these abstract synthetic modalities and real RGB frames into a shared, domain-invariant feature space.
Learnable Gated Unit (LGU)
Instead of statically merging these features, the framework passes them through a Learnable Gated Unit (LGU). The LGU dynamically weights the importance of location, visual style, and semantic/depth clues depending on the specific driving scenario before making the final crossing prediction.
Datasets
The authors evaluated their approach using both synthetic and real-world benchmarks:
- Syn-PCP-3181 (New): A large-scale synthetic dataset created by the authors using the CARLA simulator. It contains 3,181 video sequences (489,740 frames) complete with RGB, depth, semantic segmentation, and pedestrian bounding boxes across balanced weather and lighting conditions.
- JAAD & PIE: Two widely used real-world datasets captured in actual driving environments, used here to test the model's real-world generalization.
- PC50-DADA: A specialized 50-sequence subset of the DADA-2000 crash dataset, used to evaluate how well the model predicts sudden, near-collision pedestrian crossings.
Key Insights and Ideas
- Differentiated Adaptation is Key: Treating bounding boxes, raw pixels, and depth maps with the same domain-transfer technique causes "domain confusion." Tailoring the tool to the size of the gap yields cleaner feature representations.
- Intent Follows Motion: Incorporating Future Object Location (FOL) prediction as an auxiliary task significantly boosts PCP accuracy. Teaching the model where a pedestrian will physically be helps it better understand what the pedestrian intends to do.
- Naïve Mixing Fails: The authors showed that simply combining synthetic and real data into one big training bucket ("Syn+Real") actually degrades performance compared to training on real data alone. Structured, gated knowledge transfer is mandatory.
Key Results
Gated-S2R-PCP demonstrated superior performance over existing state-of-the-art (SOTA) methods:
- On the PIE Dataset: Achieved an accuracy of 0.92 and an AUC of 0.90, outperforming standard baselines like PCPA and matching or beating highly complex transformer architectures.
- Ablation Success: Fusing all three modules (KnowD + StyS + DistA) via the LGU provided the highest overall accuracy, proving that the gating mechanism successfully selects the right features at the right time.
- Near-Collision Performance: On the brutal PC50-DADA dataset—where pedestrians suddenly step into traffic—the model achieved an overall accuracy of 0.66. While lower than standard datasets, it successfully demonstrated that prediction accuracy increases as the pedestrian gets closer to the vehicle (rising to 0.63 at a 1.0-second Time-to-Crossing window).
Limitations and Future Work
Despite the framework's success, the authors note that near-collision prediction remains an open challenge. In scenarios where pedestrians appear suddenly from behind obstacles, the model's accuracy drops, indicating a massive space for improvement in rapid-reaction perception.
Additionally, while auxiliary FOL prediction helps short-term accuracy, attempting to predict pedestrian locations too far into the future (e.g., 1.0 second or more) introduces erratic behavioral noise that can actually hurt the model's recall. Future work will likely focus on tightening these prediction windows and improving robustness in occluded environments.
Summary updated