Machine learning for Internet of Things (IoT) device identification: a comparative study

Hamid Tahaei, Anqi Liu, Hamid Forooghikian, Mehdi Gheisari, Faiz Zaki, Nor Badrul Anuar, Zhaoxi Fang, Longjun HuangPeerJ 2025 · DOI

TL;DR

This paper tackles the critical security challenge of identifying IoT devices by comparing various machine learning (ML) classifiers and feature selection (FS) methods. It introduces and evaluates the Binary Grey Wolf Optimizer (BGWO) as a novel wrapper-based FS technique, demonstrating its superior ability to reduce feature dimensionality while achieving high classification accuracy, particularly for Random Forest and Neural Network models.

What Problem Does This Paper Address?

The explosive growth of Internet of Things (IoT) devices, projected to reach 500 billion by 2030, presents a significant security and privacy challenge. These devices are often resource-constrained, lacking robust security mechanisms, making them vulnerable to cyberattacks. Identifying these devices is a crucial first step in cybersecurity, and machine learning-based "device fingerprinting" using network traffic is a promising approach. However, the field lacks a comprehensive comparative analysis of different ML classifiers and feature selection techniques to understand their true effectiveness and trade-offs across various datasets. This paper aims to fill this gap by evaluating several reputable ML classifiers and FS methods, including a newly implemented BGWO, to provide a clearer picture of their capabilities for IoT device identification.

Methods

The study employs a multi-step approach:

  • Data Collection and Preprocessing: Two publicly available datasets (ProfilloT and IoT Sentinel) were used. Data was cleaned, normalized, and features were extracted to create feature vectors representing device behavior.
  • Machine Learning Classifiers: Four well-established supervised ML classifiers were evaluated: Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), and Neural Networks (NN).
  • Feature Selection (FS) Methods:
    • Filter-based: Pearson Correlation Coefficient (PCC) and Mutual Information (MI) were used to select features based on statistical properties, independent of the classifier.
  • Wrapper-based: Binary Genetic Algorithm (BGA), Binary Particle Swarm Optimization (BPSO), and the newly implemented Binary Grey Wolf Optimizer (BGWO) were employed. These methods use the classifier's performance to evaluate feature subsets, iteratively refining them. BGWO, inspired by wolf hunting behavior, uses an adaptive control mechanism to balance exploration and exploitation for feature selection.
  • Evaluation Metrics: The primary metrics were classification accuracy (TP+TN / Total) and the number of selected features. A fitness function was defined for BGWO to balance accuracy and feature reduction.

Datasets

The study utilized two real-world IoT network traffic datasets:

  • Dataset 1: ProfilIoT Dataset (Meidan et al., 2017): Contains network traffic from 8 IoT device types, with 188 features after preprocessing.
  • Dataset 2: IoT Sentinel (Aalto) Dataset (Tien et al., 2020): Contains traffic from 10 unique devices (selected from 31), with 30 features after preprocessing.

Key Insights and Ideas

  • Wrapper-based FS Outperforms Filter-based: The study consistently found that wrapper-based FS methods (BGA, BPSO, BGWO) generally achieved higher accuracy and better feature reduction compared to filter-based methods (PCC, MI).
  • BGWO is a Powerful Wrapper: The newly implemented BGWO demonstrated superior performance among the wrapper-based methods, significantly reducing feature dimensionality while achieving high classification accuracy.
  • Classifier-Specific Performance: Random Forest and Neural Networks showed notable improvements in accuracy when combined with effective FS methods, especially BGWO. Decision Trees also performed well, particularly on lower-dimensional data. SVM, while robust, tended to select more features.
  • Feature Importance Varies: The effectiveness of FS methods, particularly filter-based ones like PCC, can be highly dependent on the dataset and classifier, sometimes leading to performance degradation if relevant features are removed.
  • Dimensionality Reduction is Key: The study emphasizes that reducing the number of features is as important as achieving high accuracy, as it directly impacts classifier performance and computational efficiency.

Key Results

  • BGWO Performance: BGWO effectively reduced the feature set by 85.11% (Dataset 1) and 73.33% (Dataset 2).
  • Highest Accuracies: BGWO achieved classification accuracies of 98.51% (Dataset 1) and 99.8% (Dataset 2).
  • Random Forest and NN with BGWO: RF and NN classifiers, when paired with BGWO, showed significant accuracy improvements, with RF achieving up to 99.38% and NN up to 97.95% on Dataset 1.
  • DT and RF Robustness: DT and RF classifiers consistently performed well across various FS methods, especially on Dataset 2, demonstrating their robustness.
  • Feature Reduction: Wrapper-based methods, particularly BGWO, consistently reduced the number of features significantly without compromising accuracy. For instance, BGWO reduced 188 features to 21-37 for RF on Dataset 2.

Limitations and Future Work

  • Privacy Concerns: The paper acknowledges ongoing open issues related to data privacy in IoT.
  • Dataset Size: The authors suggest that larger datasets with more devices and features could yield more reliable outcomes and reveal deeper insights into classifier behavior.
  • Supervised Learning Focus: This study focused on supervised learning. Future work could explore combining BGWO with unsupervised and deep learning-based algorithms.
  • Current ML Trends: The authors plan to evaluate current ML trends like multimodal and federated-based learning in future research.

Summary updated

Abstract

The rapid deployment of millions of connected devices brings significant security challenges to the Internet of Things (IoT). IoT devices are typically resource-constrained and designed for specific tasks, from which new security challenges are introduced. As such, IoT device identification has garnered substantial attention and is regarded as an initial layer of cybersecurity. One of the major steps in distinguishing IoT devices involves leveraging machine learning (ML) techniques on device network flows known as device fingerprinting. Numerous studies have proposed various solutions that incorporate ML and feature selection (FS) algorithms with different degrees of accuracy. Yet, the domain needs a comparative analysis of the accuracy of different classifiers and FS algorithms to comprehend their true capabilities in various datasets. This article provides a comprehensive performance evaluation of several reputable classifiers being used in the literature. The study evaluates the efficacy of filter-and wrapper-based FS methods across various ML classifiers. Additionally, we implemented a Binary Green Wolf Optimizer (BGWO) and compared its performance with that of traditional ML classifiers to assess the potential of this binary meta-heuristic algorithm. To ensure the robustness of our findings, we evaluated the effectiveness of each classifier and FS method using two widely utilized datasets. Our experiments demonstrated that BGWO effectively reduced the feature set by 85.11% and 73.33% for datasets 1 and 2, respectively, while achieving classification accuracies of 98.51% and 99.8%, respectively. The findings of this study highlight the strong capabilities of BGWO in reducing both the feature dimensionality and accuracy gained through classification. Furthermore, it demonstrates the effectiveness of wrapper methods in the reduction of feature sets.
BibTeX
@inproceedings{tahaei2025machine,
  author = {Hamid Tahaei and Anqi Liu and Hamid Forooghikian and Mehdi Gheisari and Faiz Zaki and Nor Badrul Anuar and Zhaoxi Fang and Longjun Huang},
  title = {{Machine learning for Internet of Things (IoT) device identification: a comparative study}},
  booktitle = {PeerJ},
  year = {2025},
  doi = {10.7717/peerj-cs.2873},
}
Figure