Abstract
The integration of digital devices in modern vehicles has revolutionized automotive technology, enhancing safety and the overall driving experience. The Controller Area Network (CAN) bus is a central system for managing in-vehicle communication between the electronic control units (ECUs). However, the CAN protocol poses security challenges due to inherent vulnerabilities, lacking encryption and authentication, which, combined with an expanding attack surface, necessitates robust security measures. In response to this challenge, numerous Intrusion Detection Systems (IDS) have been developed and deployed. Nonetheless, an open, comprehensive, and realistic dataset to test the effectiveness of such IDSs remains absent in the existing literature. This paper addresses this gap by considering the latest ROAD dataset, containing stealthy and sophisticated injections. The methodology involves dataset labelling and the implementation of both state-of-the-art deep learning models and traditional machine learning models to show the discrepancy in performance between the datasets most commonly used in the literature and the ROAD dataset, a more realistic alternative.
Introduction
The Controller Area Network (CAN) bus serves as the central nervous system of modern vehicles, facilitating communication between various Electronic Control Units (ECUs). Despite its critical role, the CAN protocol was designed without native security features like encryption or authentication. As vehicles become increasingly connected, this lack of security has become a significant liability. While the research community has proposed numerous AI-driven Intrusion Detection Systems (IDS) to mitigate these risks, many of these systems have been evaluated using datasets that do not reflect real-world attack sophistication.
The Evaluation Gap: Beyond Noisy Attacks
A primary challenge in automotive cybersecurity research is the reliance on "noisy" datasets, such as the HCRL Car-Hacking dataset. These benchmarks typically feature high-intensity attacks, such as DoS or message flooding, which are relatively easy to detect because they significantly alter the timing and frequency of the bus traffic. However, sophisticated attackers are more likely to use stealthy tactics that mimic legitimate traffic patterns.
This study addresses this "evaluation gap" by utilizing the Real ORNL Automotive Dynamometer (ROAD) dataset. Unlike its predecessors, the ROAD dataset includes masquerade and targeted fabrication attacks. These attacks are "timing opaque," meaning they do not disrupt the expected frequency of messages, making them nearly invisible to traditional frequency-based detection methods.
Methodology and Model Architectures
The authors conducted a comprehensive comparative analysis between traditional machine learning (ML) and modern deep learning (DL) architectures to evaluate their effectiveness against stealthy injections.
Traditional Machine Learning
The study implemented several baseline ML models, including Random Forest, LightGBM, and the Leader Class and Confidence Decision Ensemble (LCCDE). These models are often favored for their lower computational requirements compared to neural networks.
Deep Learning Architectures
Three primary DL architectures were evaluated:
- Long Short-Term Memory (LSTM): A recurrent neural network designed to capture temporal dependencies in sequence data.
- Transformer-based Attention Network (TAN): An architecture utilizing transformer encoders to identify long-range dependencies within CAN ID sequences.
- Deep Convolutional Neural Network (DCNN): A model that treats sequences of CAN IDs as bitwise "images" to identify malicious patterns.
The DCNNv2 Innovation
During the evaluation, the authors identified a flaw in the original DCNN implementation: the use of a non-overlapping sliding window. This approach could inadvertently "cut" an attack pattern in half, leading to false negatives. To resolve this, the authors proposed DCNNv2, which utilizes a sliding window with a step of 1. This ensures the model analyzes every possible 29-message sequence, significantly improving detection sensitivity.
Dataset Comparison
The research utilized three distinct datasets to highlight the performance discrepancy across different environments:
- ROAD Dataset: The primary focus, featuring 33 attack captures from a real vehicle on a dynamometer, including 11 classes of stealthy attacks.
- HCRL Car-Hacking Dataset: The current industry benchmark, primarily featuring DoS and Fuzzy attacks.
- In-Vehicle Network IDS Challenge Dataset: Traffic captured from three stationary vehicles (Sonata, Soul, Spark).
Key Results and Insights
The Performance Gap
The study revealed a stark contrast in model performance across datasets. Models that achieved near-perfect F1-scores (99.9%) on the HCRL dataset saw their performance plummet when faced with the ROAD dataset. This suggests that high performance on older benchmarks may be a result of over-optimization for unrealistic, noisy data rather than true detection capability.
DCNNv2 Superiority
DCNNv2 emerged as the top-performing model on the ROAD dataset. By refining the sliding window approach, DCNNv2 raised the average F1-score from 0.65 (original DCNN) to 0.74. While this represents a significant improvement, it also highlights the inherent difficulty of detecting stealthy attacks.
Failure of Traditional ML
Traditional ML models like Random Forest and LightGBM proved largely ineffective against the ROAD dataset. While they performed well on simple datasets, they failed to detect masquerade attacks entirely, confirming that they are not fit for the task of securing modern vehicular networks against sophisticated threats.
Binary vs. Multiclass Detection
Deep learning models, particularly LSTM, showed exceptional performance in binary classification (detecting the presence of an attack). However, they struggled with multiclass classification—identifying the specific type of attack. This is attributed to the extreme similarity between different stealthy attack signatures and the extreme data imbalance in the ROAD dataset, where some attack classes have very few samples.
Limitations and Future Directions
Despite the success of models like DCNNv2 and TAN, their implementation in real-world vehicles faces hurdles. These models introduce significant memory and processing overhead, which may exceed the capabilities of resource-constrained ECUs. Furthermore, the scarcity of training data for specific stealthy attack types remains a bottleneck.
Future research should focus on developing hybrid models that balance accuracy with computational efficiency. Additionally, the use of transfer learning or synthetic data generation may help address the data scarcity issues identified in the ROAD dataset, leading to more robust and deployable automotive security solutions.
Summary updated