AI-Driven Intrusion Detection Systems (IDS) on the ROAD Dataset: A Comparative Analysis for Automotive Controller Area Network (CAN)

Lorenzo Guerra, Linhan Xu, Paolo Bellavista, Thomas Chapuis, Guillaume Duc, Pavlo Mozharovskyi, Van-Tam Nguyen

TL;DR

This paper tackles the critical need for robust intrusion detection systems (IDS) in automotive Controller Area Networks (CAN). By performing a comparative analysis of various AI and traditional machine learning models on the realistic ROAD dataset, the authors highlight the limitations of existing methods when faced with sophisticated, stealthy attacks, advocating for more realistic datasets for future IDS development.

What Problem Does This Paper Address?

Modern vehicles are increasingly digitized, relying on the Controller Area Network (CAN) bus for internal communication. However, the CAN protocol, designed for simplicity and speed, lacks inherent security features like encryption and authentication, making it vulnerable to attacks. While Intrusion Detection Systems (IDS) are crucial for protecting these networks, their effectiveness is heavily dependent on the datasets used for training and evaluation. Existing datasets often feature unrealistic or simplistic attacks, failing to capture the complexity of real-world threats. This paper addresses this gap by evaluating state-of-the-art AI and machine learning IDS models on the ROAD dataset, which contains stealthy and sophisticated attacks validated on a real vehicle, aiming to reveal discrepancies in performance compared to more traditional datasets.

Methods

The authors compare several AI-driven and traditional machine learning approaches for CAN intrusion detection. These include:

  • Traditional Machine Learning: Random Forest, LightGBM, and the Leader Class and Confidence Decision Ensemble (LCCDE) model. These are generally faster and more interpretable.
  • Deep Learning Models:
    • LSTM (Long Short-Term Memory): A recurrent neural network well-suited for sequential data, adapted for both binary and multiclass classification.
  • DCNN (Deep Convolutional Neural Network): A model that processes CAN traffic by building frames from sequences of IDs. A modified version (DCNNv2) was also tested with a different frame creation strategy.
  • TAN (Transformer-based Attention Network): A transformer encoder-based architecture, adapted for sequence analysis of CAN IDs.

The study involves training and testing these models on various datasets, with a particular focus on the ROAD dataset, employing both multiclass and binary classification strategies where appropriate.

Datasets

The paper utilizes and compares three datasets:

  • ROAD Dataset: This is the primary focus of the study. It contains 33 attack captures and 12 ambient captures from a real vehicle driven on a dynamometer. It features 11 distinct attack classes, including stealthy and sophisticated injections, and is considered more realistic than previous benchmarks.
  • HCRL Car-Hacking Dataset: A popular dataset generated by injecting CAN messages into a Hyundai YF Sonata. It includes four attack types.
  • In-Vehicle Network Intrusion Detection Challenge Dataset: This dataset comprises traffic data from three different vehicles, featuring three attack categories (Flooding, Fuzzy, and Malfunction).

The authors note that the ROAD dataset is imbalanced and required oversampling using the SMOTE algorithm to ensure adequate representation of minority attack classes.

Key Insights and Ideas

  • The "Realism Gap": There's a significant performance disparity between models tested on traditional, simpler datasets (like HCRL Car-Hacking) and the more realistic ROAD dataset. Models that perform near-perfectly on older datasets struggle considerably with the stealthier attacks in ROAD.
  • Stealthy Attacks are the Real Challenge: Attacks that are "Timing Transparent" (TT) and involve subtle modifications or injections of single frames (like some in the ROAD dataset) are much harder to detect than "Timing Opaque" (TO) attacks that disrupt normal timing or ID distributions.
  • Binary vs. Multiclass Classification Trade-offs: While binary classification approaches (e.g., one model per attack type) can achieve very high F1 scores on specific attacks, this is not practical for real-world IDS due to the need for numerous models and the inability to detect novel attack types. Multiclass classification is more desirable but significantly harder to achieve with high accuracy on realistic datasets.
  • Dataset Quality is Paramount: The effectiveness of an IDS is heavily reliant on the quality and realism of the training data. The ROAD dataset, with its stealthy and validated attacks, serves as a crucial benchmark for evaluating true IDS capabilities.
  • Limitations of Traditional ML: Traditional machine learning models, while fast and interpretable, are generally not sufficient for detecting the sophisticated attacks present in the ROAD dataset.

Key Results

  • Traditional ML Models (RF, LightGBM, LCCDE): Performed exceptionally well on the HCRL Car-Hacking and In-Vehicle Network Intrusion Detection Challenge datasets (often achieving near-perfect F1 scores). However, their performance on the ROAD dataset was significantly lower, with an average F1 score of 0.3212 for LCCDE, and they failed to detect some masquerade attacks.
  • LSTM: Showed strong performance in binary classification on the ROAD dataset (F1 score of 0.9977 for fuzzing attack) but struggled with multiclass classification, producing many false positives. Its performance diverged significantly between simpler and more realistic datasets.
  • DCNN: Reproduced good performance on the HCRL dataset but showed instability on the In-Vehicle Network Intrusion Detection Challenge dataset and dramatic fluctuations on the ROAD dataset. A modified version (DCNNv2) showed improvement on ROAD, but the approach still faces challenges with data field attacks and requires multiple models for different attack types.
  • TAN: Maintained good performance on most ROAD attacks but struggled with attacks having few samples. It tended to produce more false positives than false negatives.

Limitations and Future Work

  • ROAD Dataset Limitations: The ROAD dataset, while realistic, suffers from a small number of samples for some attack classes (e.g., Max Engine Coolant Temp attacks), which hinders the training of deep learning models and leads to poor performance on those specific classes. The class imbalance was addressed with SMOTE, but the underlying low sample count remains a challenge.
  • Explainability: The paper acknowledges that the explainability of the models was not examined. Understanding why an IDS makes a certain prediction is crucial for debugging, adaptation, and building trust.
  • Unseen Attacks: The study did not evaluate the models' performance against entirely new, unseen attacks, which is a critical aspect of real-world IDS evaluation.
  • Future Work: The authors suggest that future research should focus on developing new datasets that better represent realistic attack scenarios, including unseen attacks. They also emphasize the importance of model explainability and the need for IDS to be generalizable and adaptable to a wider range of threats.

Summary updated

Abstract

The integration of digital devices in modern vehicles has revolutionized automotive technology, enhancing safety and the overall driving experience. The Controller Area Network (CAN) bus is a central system for managing in-vehicle communication between the electronic control units (ECUs). However, the CAN protocol poses security challenges due to inherent vulnerabilities, lacking encryption and authentication, which, combined with an expanding attack surface, necessitates robust security measures. In response to this challenge, numerous Intrusion Detection Systems (IDS) have been developed and deployed. Nonetheless, an open, comprehensive, and realistic dataset to test the effectiveness of such IDSs remains absent in the existing literature. This paper addresses this gap by considering the latest ROAD dataset, containing stealthy and sophisticated injections. The methodology involves dataset labelling and the implementation of both state-of-the-art deep learning models and traditional machine learning models to show the discrepancy in performance between the datasets most commonly used in the literature and the ROAD dataset, a more realistic alternative.
BibTeX
@inproceedings{guerra2024ai,
  author = {Lorenzo Guerra and Linhan Xu and Paolo Bellavista and Thomas Chapuis and Guillaume Duc and Pavlo Mozharovskyi and Van-Tam Nguyen},
  title = {{AI-Driven Intrusion Detection Systems (IDS) on the ROAD Dataset: A Comparative Analysis for Automotive Controller Area Network (CAN)}},
  booktitle = {CSCS},
  year = {2024},
  doi = {10.1145/3689936.3694696},
}
Figure