On Jointly Optimizing Partial Offloading and SFC Mapping: A Cooperative Dual-Agent Deep Reinforcement Learning Approach

Xinhan Wang, Huanlai Xing, Fuhong Song, Shouxi Luo, Penglin Dai, Bowen Zhao

TL;DR

This paper tackles the complex challenge of optimizing how mobile devices (MDs) offload computation to edge servers, specifically when tasks can be split (partial offloading) and require a sequence of network functions (SFCs). They introduce a novel cooperative dual-agent deep reinforcement learning (CDADRL) approach that trains two agents to simultaneously decide how to split tasks and where to place the necessary network functions on edge servers, achieving better performance than existing methods.

What Problem Does This Paper Address?

Modern IoT applications, like AR/VR and online gaming, are increasingly computation-intensive but mobile devices have limited resources. Multi-access edge computing (MEC) offers a solution by allowing devices to offload computation to nearby edge servers. However, efficiently managing this offloading is tricky. This paper focuses on a specific scenario:

  • Partial Offloading: Instead of sending an entire task to the edge, a task can be split into two parts – one processed locally on the MD, and the other sent to the edge. These parts can be processed in parallel, potentially improving performance.
  • Network Function Virtualization (NFV) and Service Function Chains (SFCs): Many edge applications rely on a sequence of virtualized network functions (VNFs) to process data, forming a Service Function Chain (SFC). These VNFs need to be deployed on MEC servers.
  • Joint Optimization: The crucial challenge is that the decision of how much to offload (task partition) is tightly coupled with the decision of where to place the VNFs on the edge servers (SFC mapping). Optimizing one without considering the other leads to suboptimal results.

The goal is to minimize a combined cost that includes execution delay, the MD's energy consumption, and the cost of using edge computing resources.

Methods

The core of the paper is the Cooperative Dual-Agent Deep Reinforcement Learning (CDADRL) algorithm. This approach uses two agents that learn and collaborate:

  • Agent A₀ (MD Environment): This agent is responsible for task partition. It learns to decide the optimal ratio of the task to be processed locally versus offloaded to the edge. It uses a TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm, which is well-suited for continuous action spaces like offloading ratios and addresses issues like Q-value overestimation found in simpler DDPG methods.
  • Agent Aₚ (Edge Infrastructure Environment): This agent handles VNF placement. It learns to decide which edge servers should host the VNFs required for the offloaded part of the task. It employs a Dueling DDQN (Deep Q-Network) algorithm, which is effective for discrete action spaces (selecting specific servers) and improves upon standard DDQN by separating value and advantage estimation.

These two agents interact by exchanging information: Agent A₀'s offloading decision influences Agent Aₚ's environment, and Agent Aₚ's VNF placement results provide feedback (rewards) to Agent A₀. This cooperative learning allows them to jointly optimize the complex POSMJO problem.

Datasets

Not discussed in the paper. The authors use a simulated environment for their experiments.

Key Insights and Ideas

  • The Interdependence of Task Partition and VNF Placement: The paper strongly emphasizes that these two decisions are not independent. A good offloading ratio can be wasted if VNFs are poorly placed, and vice-versa. CDADRL's strength lies in its ability to learn these interdependencies.
  • TD3 for Task Partition: Using TD3 for the continuous task partitioning problem is a key choice, as it offers more stable learning and better performance compared to simpler DDPG variants by addressing overestimation.
  • Dueling DDQN for VNF Placement: For the discrete VNF placement problem, Dueling DDQN's architecture helps in learning more robust policies by better disentangling state value from action advantage.
  • Cooperative Learning is Crucial: The success of CDADRL hinges on the agents' ability to communicate and learn from each other's actions and outcomes, mimicking a realistic scenario where the MD and edge infrastructure must coordinate.

Key Results

  • Superior Performance: CDADRL consistently outperformed three other DRL algorithm combinations (TD3-DDQN, DDPG-Duel, DDPG-DDQN) in terms of cumulative and average episodic rewards.
  • Reduced Costs: Compared to baseline offloading schemes (Local, Edge, Binary, Random), CDADRL achieved significantly lower average execution delay, energy consumption, and usage charges across various MD computing capacities.
  • Faster Training: CDADRL demonstrated a faster training speed than TD3-DDQN, indicating more efficient learning.
  • Robustness: The algorithm showed good performance across different MD computing capacities, suggesting its adaptability.

Limitations and Future Work

  • Scalability: The current work focuses on a single MD. The authors suggest extending the algorithm to handle large-scale networks with multiple MDs requesting offloading and SFC mapping services.
  • Multi-Objective Optimization: The paper aggregates multiple objectives (delay, energy, cost) into a single objective function using weighted sums. However, the authors note that in some real-world scenarios, these weights might be unknown or change dynamically. They propose extending their algorithm to a multi-objective RL method to handle such situations.
  • Dynamic Weights: The current approach assumes fixed weights for the objective functions. Future work could explore how to adapt these weights dynamically.

Summary updated

Abstract

Multi-access edge computing (MEC) and network function virtualization (NFV) are promising technologies to support emerging IoT applications, especially those computation-intensive. In NFV-enabled MEC environment, service function chain (SFC), i.e., a set of ordered virtual network functions (VNFs), can be mapped on MEC servers. Mobile devices (MDs) can offload computation-intensive applications, which can be represented by SFCs, fully or partially to MEC servers for remote execution. This article studies the partial offloading and SFC mapping joint optimization (POSMJO) problem in an NFV-enabled MEC system, where the data from an incoming task is partitioned into two parts, with one part executed locally and the other offloaded to the edge infrastructure for execution. These two parts are independent of each other, but both need to be processed by the same SFC. The objective is to minimize the average cost in the long term which is a combination of execution delay, MD's energy consumption, and usage charge for edge computing. This problem consists of two closely related decision-making steps, namely task partition and VNF placement, which is highly complex and quite challenging. To address this, we propose a cooperative dual-agent deep reinforcement learning (CDADRL) algorithm, where two agents interact with each other. Simulation results show that the proposed algorithm outperforms three combinations of deep reinforcement learning algorithms with respect to cumulative reward and it overweighs a number of baseline algorithms in terms of execution delay, energy consumption, and usage charge.
BibTeX
@article{wang2023jointly,
  author = {Xinhan Wang and Huanlai Xing and Fuhong Song and Shouxi Luo and Penglin Dai and Bowen Zhao},
  title = {{On Jointly Optimizing Partial Offloading and SFC Mapping: A Cooperative Dual-Agent Deep Reinforcement Learning Approach}},
  journal = {IEEE Trans. Parallel Distributed Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=QszINIJ0C4},
}
Figure