Comparative DQN-Improved Algorithms for Stochastic Games-Based Automated Edge Intelligence-Enabled IoT Malware Spread-Suppression Strategies

Yizhou Shen, Carlton Shepherd, Chuadhry Mujeeb Ahmed, Shui Yu, Tingting LiIEEE Internet of Things Journal 2024 · DOI

The proliferation of Internet of Things (IoT) devices has created a massive attack surface for rapidly evolving malware. Traditional centralized security measures often suffer from high latency and resource constraints. This research, published in the IEEE Internet of Things Journal, proposes a decentralized approach using Edge Intelligence-enabled IoT (EIIoT) to autonomously suppress malware spread through advanced reinforcement learning.

The Challenge of IoT Malware Suppression

Modern IoT malware evolves "sharply," making static patching and manual analysis ineffective. Furthermore, these environments are characterized by "incomplete information," where neither the defender nor the attacker has full visibility into the other's state. To address this, the authors propose moving the defensive "brain" from the cloud to edge nodes, enabling localized, low-latency autonomous responses.

The Edge Intelligence-enabled IoT (EIIoT) architecture, showing the hierarchy from cloud servers to edge nodes and IoT end devices.

Stochastic Games and Edge Intelligence

The core of the proposed solution is the Stochastic Games-based Malware Spread-Suppression (SGMSS) model. This mathematical framework treats the interaction between malware and edge defenders as a multi-agent, dynamic game with probabilistic state transitions. The goal is to find a Nash equilibrium—a stable strategy where neither party can improve their outcome by unilaterally changing their approach.

To solve this game, the researchers implemented three variants of Deep Q-Networks (DQN) designed to overcome common pitfalls like value overestimation:

  • DDQMS (Double DQN for Malware Spread): Decouples action selection from value estimation using two separate networks, leading to more accurate Q-values.
  • D2QMS (Dueling DQN for Malware Spread): Splits the network's final layer to independently estimate state value and action advantage, improving learning in states where actions have delayed impacts.
  • D3QMS (Dueling Double DQN for Malware Spread): A hybrid architecture combining both Double and Dueling DQN features for maximum stability and performance.
The training workflow for the malware spread-suppression agent, illustrating the interaction between the Q-networks, experience replay buffer, and the EIIoT environment.

Experimental Framework

The algorithms were evaluated using idsgame, an OpenAI Gym environment for modeling stochastic games in cybersecurity. The environment simulates a Partially Observable Markov Decision Process (POMDP). In these simulations, malware followed a random spread policy while edge nodes were trained using a Fuzzy Neural Network (FNN) with two hidden layers and ReLU activation.

Key Insights and Performance Results

The study yielded several critical findings for autonomous network defense:

  • Superiority of D3QMS: The hybrid D3QMS algorithm emerged as the top performer, achieving the highest cumulative rewards and the fastest reduction in average episode loss (dropping from 0.0042 to 0.0008).
  • Effective Suppression: All three proposed algorithms successfully maintained the malware spread rate below 0.15, demonstrating the viability of autonomous edge-based defense.
  • The "Short-Sighted" Advantage: Counterintuitively, a lower discount factor ($\gamma = 0.85$) outperformed the standard $\gamma = 0.99$. In highly volatile malware environments, prioritizing immediate suppression results leads to more robust long-term defense.
  • Optimal Hyperparameters: The researchers identified a learning rate ($\alpha$) of 4e-4 and a discount factor of 0.85 as the "sweet spot" for training these agents.

Future Directions

While the results demonstrate the power of edge intelligence, the authors note that future work should test these defenses against "intelligent" attackers using their own reinforcement learning strategies. Additionally, further research is needed to quantify the computational overhead on resource-constrained IoT hardware and to evaluate performance in larger, more complex network topologies.

Summary updated

BibTeX
@article{shen2024comparative,
  author = {Yizhou Shen and Carlton Shepherd and Chuadhry Mujeeb Ahmed and Shui Yu and Tingting Li},
  title = {{Comparative DQN-Improved Algorithms for Stochastic Games-Based Automated Edge Intelligence-Enabled IoT Malware Spread-Suppression Strategies}},
  journal = {IEEE Internet of Things Journal},
  year = {2024},
  doi = {10.1109/jiot.2024.3381281},
}
Figure