TL;DR
This paper presents a novel monocular vision-based approach for detecting pedestrian intentions, specifically their likelihood of crossing the road. By analyzing a pedestrian's pose over time using a CNN-based skeleton estimator, the system can anticipate crossing actions up to 750ms in advance, offering crucial reaction time for ADAS and autonomous vehicles.
What Problem Does This Paper Address?
Vehicle-to-pedestrian crashes are a major concern for advanced driver assistance systems (ADAS) and autonomous vehicles. While detecting pedestrians has seen significant advancements with deep learning, a more critical challenge is predicting their intentions. Simply knowing a pedestrian is present isn't enough; understanding if they are about to step into the vehicle's path is essential for safe and comfortable maneuvers. Existing research in pedestrian intention detection is less developed than pedestrian detection itself, and many approaches rely on complex sensor setups like stereo vision or optical flow. This paper aims to bridge this gap by developing a more accessible, vision-only method for predicting pedestrian intentions.
Methods
The proposed approach is a multi-stage pipeline:
- Pedestrian Detection and Tracking: The system starts with a 2D bounding box for each pedestrian, obtained from standard detection and tracking methods. The authors are agnostic to the specific algorithms used here.
- 2D Pose Estimation: A state-of-the-art CNN-based method (specifically, a two-branch multi-stage CNN trained on COCO) is employed to fit a 2D skeleton to the pedestrian within the bounding box. This step is robust to minor inaccuracies in the bounding box and pedestrian scale.
- Skeleton Feature Extraction: From the fitted skeleton, a feature vector is computed. The authors focus on nine keypoints (legs and shoulders) deemed most stable and informative. These keypoints are normalized by the pedestrian's estimated height. Features include distances, relative angles between keypoints, and triangle angles.
- Temporal Feature Concatenation: To capture the dynamic nature of intentions, feature vectors from the last
Tframes (here,T=10) are concatenated to form a temporal feature vector for each pedestrian. - Classifier: A binary classifier (either Random Forest or SVM) is trained to predict specific intentions (e.g., crossing vs. stopping).
A key innovation is that this method is monocular, meaning it only requires a single camera feed and does not rely on stereo vision or optical flow, making it more practical for implementation.
Datasets
The authors utilize a publicly available dataset introduced in [7] and previously used in [11, 12]. This dataset contains sequences recorded from a forward-facing windshield camera at 16 FPS, with resolutions of 1176x640 pixels. It includes ground truth annotations for pedestrian intentions (stopping, crossing, bending, starting) and time-to-event (TTE) information. The dataset is split into training and testing sets.
Key Insights and Ideas
- Pose as a Rich Indicator: Analyzing the detailed pose of a pedestrian over time, rather than just their bounding box or overall motion, provides significantly more information about their intentions. The movement of legs and the orientation of the upper body are crucial cues.
- Monocular Vision Sufficiency: The success of the approach demonstrates that sophisticated intention prediction can be achieved with a single camera, reducing the complexity and cost of sensor suites for ADAS.
- Temporal Context is Crucial: Concatenating pose features over multiple frames (a temporal sliding window) is vital for capturing the subtle, evolving signals that indicate an intention to cross or perform another action.
- Human-like Anticipation: The system achieves anticipation times comparable to or better than reported human reaction times in some scenarios, suggesting a high level of predictive capability.
Key Results
- Anticipation Time: The system achieves approximately 750 ms of anticipation for pedestrians crossing the road. At 50 km/h, this translates to about 15 additional meters of reaction space.
- Crossing vs. Stopping: The method demonstrates high predictability for distinguishing between crossing and stopping actions, achieving 0.8 predictability at TTE = 12 frames (approximately 750 ms before the event) when using ground truth bounding boxes. Similar results were obtained using detected bounding boxes.
- Bending: For bending actions, the system achieves 0.8 predictability at TTE = -2 frames (125 ms after the event) with GT bounding boxes, and TTE = -4 frames (250 ms after the event) with detected bounding boxes. The authors note this is more about understanding the action is happening rather than predicting it far in advance.
- Starting: For starting actions, 0.8 predictability is achieved at TTE = 3 frames (187 ms) for GT bounding boxes and TTE = 3 frames (187 ms) for detected bounding boxes.
- Comparison to State-of-the-Art: The proposed method shows comparable or better results than previous work [11, 12] while using a simpler, monocular setup.
Limitations and Future Work
- Far Pedestrians and Viewpoint Ambiguity: The system struggles with pedestrians at far distances, particularly when viewed from the front or back. This leads to instability in pose estimation and affects the accuracy of intention detection, especially for bending.
- Dataset Limitations: The current dataset has limited sequences for certain intentions (e.g., "starting"), necessitating data augmentation strategies.
- Occlusion and Multiple Pedestrians: The paper acknowledges that future work should address scenarios with multiple, potentially occluding pedestrians, which would require new datasets.
- Head Pose Robustness: While head pose is a potential cue, its detection is not as robust as other body parts, and its utility for intention detection needs further investigation.
- Future Directions: The authors suggest exploring head orientation as an additional cue, improving robustness to viewpoint ambiguity, and developing datasets for more complex multi-pedestrian scenarios. They also mention that while RF performed worse for some tasks, it's not a general conclusion and depends on the specific dataset and training.
Summary updated