TL;DR
This paper investigates the hyper-parameter tuning of a deep learning model for IoT intrusion detection, revealing that the original reported accuracies are likely overstated. By systematically exploring hyper-parameter effects, the authors demonstrate that achieving the claimed performance is not possible with the described methods, suggesting the original paper may have conflated training and testing accuracies.
What Problem Does This Paper Address?
The Internet of Things (IoT) is rapidly expanding, creating a vast attack surface. Detecting intrusions in such distributed environments is crucial, and deep learning models have shown promise. One such model, proposed by Diro and Chilamkurti [1], claimed exceptionally high accuracy on the NSL-KDD dataset for intrusion detection. However, the reproducibility of these results is questionable, especially given that a subsequent study [2] reported significantly lower accuracies for similar models on the same dataset. This paper aims to replicate the original study's findings and, more importantly, to understand the impact of hyper-parameter choices on the model's performance, thereby clarifying the true capabilities of the proposed deep learning approach.
Methods
The authors adopted the pre-processing steps, network architecture, and some parameter values from the original study [1]. The network architecture consists of fully connected sequential layers (ANN) with 122 input features, three hidden layers (150, 120, and 50 neurons respectively), and a softmax output layer. The input features are categorical and encoded using a 1-to-n technique, and then normalized using z-score normalization.
The core of this paper's methodology lies in its rigorous hyper-parameter optimization. While the original paper [1] mentioned hyper-parameters like activation function, loss function, optimizer, mini-batch size, epochs, learning rate, and dropout rates, specific values for achieving the reported accuracy were not provided. This study systematically varied key hyper-parameters, including:
- Learning Rate: Explored across a range from 0.001 to 0.2.
- Batch Size: Tested at 8, 16, 32, 48, and 64.
- Dropout Rate: Varied for both the input layer (fixed at 20%) and hidden layers (20%, 30%, 40%, 50%).
The authors used the rectified linear unit (ReLU) for hidden layers and softmax for the output layer, with mean squared error as the loss function and SGD as the optimizer, consistent with the original paper's stated choices where available.
Datasets
The primary dataset used for training and evaluation in this study is the NSL-KDD dataset. This dataset is a refined version of the KDD Cup 99 dataset, designed to address some of its limitations. The authors evaluated the model's performance on both:
- 2-class classification: Distinguishing between "Normal" and "Attack" traffic.
- 4-class classification: Categorizing attacks into "Normal," "DoS," "Probe," and "R2L.U2R."
Key Insights and Ideas
- Reproducibility Crisis in Action: The most striking insight is the stark contrast between the claimed accuracies in [1] and the achievable accuracies through systematic hyper-parameter optimization. This highlights a common challenge in research: the difficulty of reproducing reported results, especially when crucial details like hyper-parameter settings are omitted.
- Overstated Capabilities: The paper strongly suggests that the original paper [1] likely overstated the model's capabilities, potentially by reporting training accuracy instead of test accuracy. This is a critical distinction for evaluating a model's generalization performance.
- Hyper-parameter Sensitivity: The study identifies that certain hyper-parameters have a more significant impact on performance than others. While not explicitly detailed which three are most influential, the extensive experimentation implies a non-trivial relationship between these settings and the final accuracy.
- Validation Against Prior Work: The findings align more closely with the results reported by Yin et al. [2], reinforcing the notion that the accuracies in [1] were unusually high and potentially misreported.
Key Results
The study found that the best achievable test accuracy for the 2-class classification on the NSL-KDD dataset was 84.21%. This was achieved with a learning rate of 0.1, a batch size of 48, and a dropout rate of 20% for the input layer and 20% for the hidden layers.
For the 4-class classification, the best test accuracy obtained was 80.95%. This was achieved with a learning rate of 0.1, a batch size of 64, and a dropout rate of 20% for the input layer and 30% for the hidden layers.
These results are significantly lower than the 99.20% (2-class) and 98.27% (4-class) reported in the original paper [1]. The authors note that their achieved accuracies are comparable to the training accuracies reported in [1] and [2], further supporting their hypothesis of misreporting.
Limitations and Future Work
Limitations:
- Absence of Original Code: The primary limitation faced by the authors was the absence of the source code and specific hyper-parameter values from the original study [1], necessitating a replication and optimization process from scratch.
- Limited Hyper-parameter Search Space: While extensive, the hyper-parameter search space was not exhaustive. There might be other combinations or ranges that could yield slightly different results.
Future Work:
The paper implicitly suggests that future work should focus on accurate reporting of experimental results, particularly distinguishing between training and testing accuracies. It also highlights the need for robust hyper-parameter tuning to ensure the true performance of deep learning models in critical applications like IoT security. Further research could explore more advanced optimization techniques or different network architectures to potentially improve upon the achieved accuracies.
Summary updated