TL;DR
This paper tackles the complex challenge of optimizing how mobile devices (MDs) offload computation to edge servers, specifically when tasks can be split (partial offloading) and require a sequence of network functions (SFCs). They introduce a novel cooperative dual-agent deep reinforcement learning (CDADRL) approach that trains two agents to simultaneously decide how to split tasks and where to place the necessary network functions on edge servers, achieving better performance than existing methods.
What Problem Does This Paper Address?
Modern IoT applications, like AR/VR and online gaming, are increasingly computation-intensive but mobile devices have limited resources. Multi-access edge computing (MEC) offers a solution by allowing devices to offload computation to nearby edge servers. However, efficiently managing this offloading is tricky. This paper focuses on a specific scenario:
- Partial Offloading: Instead of sending an entire task to the edge, a task can be split into two parts – one processed locally on the MD, and the other sent to the edge. These parts can be processed in parallel, potentially improving performance.
- Network Function Virtualization (NFV) and Service Function Chains (SFCs): Many edge applications rely on a sequence of virtualized network functions (VNFs) to process data, forming a Service Function Chain (SFC). These VNFs need to be deployed on MEC servers.
- Joint Optimization: The crucial challenge is that the decision of how much to offload (task partition) is tightly coupled with the decision of where to place the VNFs on the edge servers (SFC mapping). Optimizing one without considering the other leads to suboptimal results.
The goal is to minimize a combined cost that includes execution delay, the MD's energy consumption, and the cost of using edge computing resources.
Methods
The core of the paper is the Cooperative Dual-Agent Deep Reinforcement Learning (CDADRL) algorithm. This approach uses two agents that learn and collaborate:
- Agent A₀ (MD Environment): This agent is responsible for task partition. It learns to decide the optimal ratio of the task to be processed locally versus offloaded to the edge. It uses a TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm, which is well-suited for continuous action spaces like offloading ratios and addresses issues like Q-value overestimation found in simpler DDPG methods.
- Agent Aₚ (Edge Infrastructure Environment): This agent handles VNF placement. It learns to decide which edge servers should host the VNFs required for the offloaded part of the task. It employs a Dueling DDQN (Deep Q-Network) algorithm, which is effective for discrete action spaces (selecting specific servers) and improves upon standard DDQN by separating value and advantage estimation.
These two agents interact by exchanging information: Agent A₀'s offloading decision influences Agent Aₚ's environment, and Agent Aₚ's VNF placement results provide feedback (rewards) to Agent A₀. This cooperative learning allows them to jointly optimize the complex POSMJO problem.
Datasets
Not discussed in the paper. The authors use a simulated environment for their experiments.
Key Insights and Ideas
- The Interdependence of Task Partition and VNF Placement: The paper strongly emphasizes that these two decisions are not independent. A good offloading ratio can be wasted if VNFs are poorly placed, and vice-versa. CDADRL's strength lies in its ability to learn these interdependencies.
- TD3 for Task Partition: Using TD3 for the continuous task partitioning problem is a key choice, as it offers more stable learning and better performance compared to simpler DDPG variants by addressing overestimation.
- Dueling DDQN for VNF Placement: For the discrete VNF placement problem, Dueling DDQN's architecture helps in learning more robust policies by better disentangling state value from action advantage.
- Cooperative Learning is Crucial: The success of CDADRL hinges on the agents' ability to communicate and learn from each other's actions and outcomes, mimicking a realistic scenario where the MD and edge infrastructure must coordinate.
Key Results
- Superior Performance: CDADRL consistently outperformed three other DRL algorithm combinations (TD3-DDQN, DDPG-Duel, DDPG-DDQN) in terms of cumulative and average episodic rewards.
- Reduced Costs: Compared to baseline offloading schemes (Local, Edge, Binary, Random), CDADRL achieved significantly lower average execution delay, energy consumption, and usage charges across various MD computing capacities.
- Faster Training: CDADRL demonstrated a faster training speed than TD3-DDQN, indicating more efficient learning.
- Robustness: The algorithm showed good performance across different MD computing capacities, suggesting its adaptability.
Limitations and Future Work
- Scalability: The current work focuses on a single MD. The authors suggest extending the algorithm to handle large-scale networks with multiple MDs requesting offloading and SFC mapping services.
- Multi-Objective Optimization: The paper aggregates multiple objectives (delay, energy, cost) into a single objective function using weighted sums. However, the authors note that in some real-world scenarios, these weights might be unknown or change dynamically. They propose extending their algorithm to a multi-objective RL method to handle such situations.
- Dynamic Weights: The current approach assumes fixed weights for the objective functions. Future work could explore how to adapt these weights dynamically.
Summary updated