Reinforcement Learning Controlled Adaptive PSO for Task Offloading in IIoT Edge Computing

Minod Perera, Sheik Mohammad Mostakim Fattah, Sajib Mistry, Aneesh Krishna

TL;DR

This paper tackles the challenge of efficiently offloading tasks in Industrial Internet of Things (IIoT) environments by combining Reinforcement Learning (RL) with an adaptive Particle Swarm Optimization (PSO) algorithm. Their novel APSO-SAC approach dynamically tunes PSO parameters, leading to significantly improved task offloading efficiency, reduced latency, and lower computational costs compared to existing methods.

What Problem Does This Paper Address?

The Industrial Internet of Things (IIoT) is generating massive amounts of data, requiring low-latency processing for applications like smart factories. Traditional cloud computing introduces significant delays. Mobile Edge Computing (MEC) brings computation closer to devices, but efficiently deciding where to offload tasks is complex. Factors like server load, environmental changes, and device capabilities make this a dynamic optimization problem. Simple strategies like always choosing the nearest server are suboptimal. While Particle Swarm Optimization (PSO) is good for static environments, it struggles with dynamic changes. Reinforcement Learning (RL) is promising for dynamic environments, but existing RL methods might not be optimal for the specific constraints of IIoT MEC. This paper aims to bridge this gap by creating a hybrid approach that leverages the strengths of both PSO and RL for more intelligent and adaptive task offloading.

Methods

The core of this paper is the proposed APSO-SAC (Adaptive Particle Swarm Optimization with Soft Actor Critic) method.

  • Particle Swarm Optimization (PSO): This is a population-based optimization algorithm where "particles" (potential solutions) move through a search space. Standard PSO has fixed hyperparameters that can hinder performance in dynamic environments.
  • Adaptive PSO (APSO): This is an extension of PSO that dynamically adjusts its hyperparameters (like acceleration coefficients and inertia weights) during the optimization process. This allows it to better transition between exploration (searching new areas) and exploitation (refining existing solutions).
  • Soft Actor Critic (SAC): This is an off-policy RL algorithm known for its ability to learn optimal policies in dynamic environments. SAC balances exploration and exploitation by maximizing expected rewards while also encouraging exploration through an entropy bonus. It's chosen for its robustness to hyperparameter tuning.

The novel contribution is the integration of SAC with APSO. The SAC agent observes the performance of the APSO algorithm and adaptively fine-tunes its hyperparameters. This allows the system to respond to environmental changes and optimize task offloading decisions in real-time without requiring extensive manual tuning or domain knowledge.

Datasets

The paper does not explicitly mention using pre-existing datasets for training or evaluation. Instead, they describe a simulated IIoT environment created using Python and Matlab.

  • Simulation Setup: The simulation involves 250 devices and 20 MEC servers.
  • Parameter Generation: Devices are characterized by data size, computational workload, and RAM requirements. Servers have processing speed, cost per unit time, and available RAM. These parameters are randomly generated within realistic ranges to mimic a diverse set of devices and servers.
  • Task Offloading Configuration: The "cost" of a task offloading configuration is evaluated based on a weighted sum of computational cost and latency.

Key Insights and Ideas

  • Hybridization is Key: The paper strongly suggests that combining swarm intelligence (PSO) with the adaptive learning capabilities of RL (SAC) is crucial for tackling the complexity of dynamic IIoT MEC environments.
  • Dynamic Adaptation Outperforms Static: The results clearly show that adaptive methods (APSO and APSO-SAC) significantly outperform standard PSO, highlighting the need for dynamic parameter tuning in real-world scenarios.
  • RL for Meta-Optimization: Using RL (SAC) as a "meta-optimizer" to tune the parameters of another optimization algorithm (PSO) is a powerful strategy for achieving robust performance across varying conditions.
  • Balancing Exploration and Exploitation: The entropy bonus in SAC is instrumental in ensuring sufficient exploration, which is vital for discovering optimal offloading strategies in a complex, dynamic search space.

Key Results

The experimental results demonstrate the effectiveness of the proposed APSO-SAC method:

  • Baseline PSO: Achieved an average best cost of 81.69.
  • Adaptive PSO (APSO): Improved the average best cost to 76.39 (a 6.5% reduction). However, this came with a significant increase in runtime (up to 110%), nullifying the benefits.
  • APSO-SAC (Proposed Method): Achieved an average best cost of 58.55, representing a substantial 28.38% improvement over baseline PSO. Crucially, this was achieved while maintaining a similar runtime to baseline PSO, with the only caveat being the initial training period for the SAC agent.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that further tuning of SAC hyperparameters could potentially yield even better results, but this would require more domain knowledge and extensive testing.
  • The current simulation environment is described as "mostly static device and server placements," though the method is designed for dynamic variables.
  • The runtime increase for APSO, while not present in APSO-SAC, is a notable limitation of adaptive methods without RL integration.
  • Future Work:
    • Investigate tuning SAC hyperparameters for potentially superior performance.
  • Explore discrete environmental models and other RL algorithms like Proximal Policy Optimization (PPO) or Advantage Actor-Critic (A2C) for smaller subsections of the problem.
  • Enhance the simulator for finer tuning and specific advancements to improve the method's applicability.

Summary updated

Abstract

Industrial Internet of Things (IIoT) applications demand efficient task offloading to handle heavy data loads with minimal latency. Mobile Edge Computing (MEC) brings computation closer to devices to reduce latency and server load, optimal performance requires advanced optimization techniques. We propose a novel solution combining Adaptive Particle Swarm Optimization (APSO) with Reinforcement Learning, specifically Soft Actor Critic (SAC), to enhance task offloading decisions in MEC environments. This hybrid approach leverages swarm intelligence and predictive models to adapt to dynamic variables such as human interactions and environmental changes. Our method improves resource management and service quality, achieving optimal task offloading and resource distribution in IIoT edge computing.
BibTeX
@inproceedings{perera2025reinforcement,
  author = {Minod Perera and Sheik Mohammad Mostakim Fattah and Sajib Mistry and Aneesh Krishna},
  title = {{Reinforcement Learning Controlled Adaptive PSO for Task Offloading in IIoT Edge Computing}},
  booktitle = {The Web Conference},
  year = {2025},
  url = {https://openreview.net/forum?id=8AabMVdQfj},
}
Figure