Introduction: The Challenge of Pedestrian Intent
Pedestrian Crossing Prediction (PCP) is a critical component for safe autonomous driving. Beyond simple detection, a vehicle must anticipate a pedestrian's intent to cross the road seconds before the action occurs. However, training robust models is hindered by the scarcity of diverse, high-risk real-world scenarios. While synthetic data from simulators like CARLA offers a solution, it introduces a significant "Sim-to-Real" domain gap.
Existing methods often apply a uniform adaptation strategy across all data types. The authors of this work observe that different modalities—pedestrian coordinates, RGB images, and depth/semantic maps—exhibit distinct domain gaps. For instance, spatial coordinates are relatively consistent between simulation and reality, whereas visual textures differ vastly.
The Gated-S2R-PCP Framework
To address these varying gaps, the researchers developed Gated-S2R-PCP, a framework employing a "differentiated" transfer approach. The system processes three distinct data pathways, each optimized for its specific modality:
- Knowledge Distiller (KnowD): Handles pedestrian location data (bounding boxes). It utilizes a teacher-student architecture where a teacher model, pre-trained on synthetic data, guides a student model learning from real-world data. This pathway also incorporates a Future Object Location (FOL) task, improving intent prediction by forecasting the pedestrian's trajectory.
- Style Shifter (StyS): Processes RGB camera frames. By using feature-level style transfer (AdaIN), it aligns the "style" of synthetic images with real-world visual characteristics like lighting and weather.
- Distribution Approximator (DistA): Manages semantic and depth maps, which typically have the largest domain gaps. It uses adversarial training to align features into a shared space.
At the core of the framework is the Learnable Gated Unit (LGU). This component acts as an adaptive conductor, weighting the contributions of the three pathways based on the scene's context. For example, in low-visibility conditions like heavy rain, the LGU may prioritize reliable location data over noisy visual features.
Datasets and Benchmarking
The study introduces Syn-PCP-3181, a massive synthetic benchmark containing nearly 490,000 frames with multi-modal data (RGB, depth, semantic) across diverse conditions. For real-world validation, the authors used the JAAD and PIE datasets, as well as DADA-2000 for near-collision scenarios.
Key Results and Insights
The Gated-S2R-PCP model achieved state-of-the-art (SOTA) performance across multiple benchmarks:
- PIE Dataset: Reached an accuracy of 0.92 and an F1-score of 0.82, outperforming previous leaders like Pedformer.
- JAAD Dataset: Demonstrated superior performance, particularly on the behavioral subset (JAAD-beh).
- Robustness: The differentiated transfer approach proved essential; treating all modalities identically was shown to degrade performance.
Conclusion and Future Directions
By acknowledging and specifically addressing the unique domain gaps of different data modalities, Gated-S2R-PCP sets a new standard for pedestrian intent prediction. While highly effective, the model currently relies on external detection and tracking systems. Future work will focus on integrating low-level perception and improving reaction times for sudden, short-range pedestrian movements in crowded urban environments.
Summary updated