TL;DR
This paper introduces a novel, lightweight, and real-time system for recognizing dangerous driving actions using 3D skeletal data. By combining a dynamic keypoint selection method (GSFR) with a Broad Learning System (BLS) and a dual smoothing strategy, the approach achieves high accuracy and robustness, making it suitable for resource-constrained driver monitoring systems.
What Problem Does This Paper Address?
Modern vehicles are increasingly equipped with Driver Monitoring Systems (DMSs) to enhance road safety by detecting dangerous driving behaviors like fatigue or distraction. However, deploying these systems on embedded devices presents significant challenges: they need to be lightweight, perform in real-time, and remain robust to noisy or incomplete data. Traditional methods often struggle with these constraints, especially when processing raw visual data. While skeletal data offers advantages in efficiency and robustness, existing skeletal-based methods still face hurdles in achieving both high accuracy and real-time performance on limited hardware. This paper aims to bridge this gap by proposing a method that is efficient, accurate, and robust for dangerous driving action recognition on embedded DMS.
Methods
The proposed approach, named GSFR-BLS, integrates several key techniques:
- Graph Spatio-Temporal Feature Representation (GSFR): This method dynamically selects the most relevant 3D skeletal keypoints by analyzing their movement over time. Instead of using all keypoints, it focuses on those exhibiting significant motion, which are more indicative of dangerous actions. This dynamic selection improves robustness to noise and reduces computational complexity.
- Broad Learning System (BLS): A lightweight neural network architecture is employed to process the selected skeletal features. The BLS is optimized with:
- Sparse Feature Selection: L1 regularization is used to enforce sparsity and retain only the most informative features.
- Principal Component Analysis (PCA): This technique further reduces the dimensionality of the selected features, enhancing efficiency without significant loss of information.
- Dual Smoothing Strategy: To stabilize predictions and reduce sensitivity to short-term fluctuations, a two-stage smoothing process is applied:
- Sliding Window Smoothing: Averages classification outputs over a window of consecutive frames.
- Exponential Moving Average (EMA): Further smooths the results to capture long-term trends and reduce jitter.
Datasets
The study utilizes two public datasets for training and evaluation:
- State Farm Dataset: This dataset, originating from a Kaggle competition, contains images and videos from in-car cameras, labeled with various driving behaviors. It's commonly used for distracted driving analysis.
- Driver Skeleton Dataset: This dataset focuses on capturing 3D skeletal data of drivers, enabling the analysis of subtle movements and postures.
The paper also mentions the use of BlazePose for obtaining 3D skeleton keypoints from images, with a data cleaning step using kNN to remove outliers.
Key Insights and Ideas
- Dynamic Keypoint Selection is Crucial: Focusing on moving keypoints, rather than static ones, significantly improves the ability to detect dynamic and potentially dangerous driving actions. This is a key insight for making skeletal data analysis more efficient and relevant.
- Broad Learning for Efficiency: The BLS architecture proves to be a powerful yet lightweight alternative to traditional deep learning models, offering a good balance between performance and computational cost, which is essential for embedded systems.
- Smoothing for Robustness: The dual smoothing approach effectively tackles the inherent noise in real-world sensor data and stabilizes predictions, leading to more reliable action recognition.
- Balancing Accuracy and Efficiency: The GSFR-BLS method demonstrates that it's possible to achieve competitive accuracy without the massive parameter counts or computational demands of some state-of-the-art methods.
Key Results
- Accuracy Improvement: The GSFR-BLS model achieved an accuracy of 93.639% on the State Farm dataset and 93.512% on the Driver Skeleton dataset, showing significant improvements over baseline BLS and other methods.
- Efficiency: The proposed method boasts a compact model size (507 KB) and a short inference time, making it highly suitable for deployment on resource-constrained embedded devices. For instance, on an NVIDIA Jetson TX2, it can reach 26.67 FPS.
- Robustness: Ablation studies showed that incorporating GSFR consistently improved accuracy across different classifiers (BLS, DNN, GCN). The smoothing strategy also demonstrated a significant reduction in noise in the classification results.
- Performance Comparison: The GSFR-BLS method outperformed or was competitive with other mainstream classifiers (Random Forest, Gradient Boosting, SVM, DNN, 1D-CNN, GCN) and state-of-the-art approaches like SC-GCN and DDDS in terms of accuracy, inference time, and parameter count.
Limitations and Future Work
- Pose Estimation Accuracy: The paper acknowledges that the model's performance is influenced by the accuracy of the initial human pose estimation.
- Data Imbalance: Potential data imbalance issues in datasets can affect model performance.
- Future Work:
- Incorporating Raw Image Data: Combining skeletal information with raw image data could further enhance classification accuracy and robustness.
- Diverse Driving Scenarios: Expanding the system's applicability by collecting data from a broader range of real-world driving scenarios, including urban/rural settings, varying age groups, and experience levels.
- Edge-Cloud Collaboration: Exploring edge-cloud collaborative DMS technologies and integrating with Vehicle-to-Everything (V2X) communication and Unmanned Aerial Vehicles (UAVs).
Summary updated