AQMDRL: Automatic Quality of Service Architecture Based on Multistep Deep Reinforcement Learning in Software-Defined Networking

Junyan Chen, Cenhuishan Liao, Yong Wang, Lei Jin, Xiaoye Lu, Xiaolan Xie, Rui YaoSensors 2022 · DOI

TL;DR

This paper introduces AQMDRL, a novel Quality of Service (QoS) architecture for Software-Defined Networking (SDN) that leverages a multistep Deep Reinforcement Learning (DRL) approach. By addressing the overestimation and underestimation issues inherent in traditional DRL algorithms like DDPG, AQMDRL significantly improves convergence speed and stability, leading to reduced network latency and better load balancing.

What Problem Does This Paper Address?

Modern data center networks, especially those supporting the Internet of Things (IoT), face immense pressure from rapidly growing traffic and stringent Quality of Service (QoS) demands. Software-Defined Networking (SDN) offers a promising solution by centralizing network control and enabling intelligent management. While Deep Reinforcement Learning (DRL) has shown potential for optimizing SDN performance (e.g., routing, scheduling), existing DRL algorithms often suffer from slow convergence and instability. This leads to a prolonged period of suboptimal network performance (jitter) before the system stabilizes, negatively impacting QoS. The authors aim to bridge this gap by developing a DRL-based architecture that achieves faster and more stable QoS optimization in SDN.

Methods

The core of this work is the AQMDRL (Automatic Quality of Service Architecture Based on Multistep Deep Reinforcement Learning). It builds upon the Deep Deterministic Policy Gradient (DDPG) algorithm but introduces two key enhancements:

  • Multistep DRL: Instead of relying on a single-step Q-value for updates, AQMDRL uses an n-step approach. This means it considers the accumulated reward and future states over multiple steps to estimate the target Q-value. This helps to mitigate both the overestimation (by using a more robust estimate) and underestimation (by incorporating future rewards) problems that plague standard DDPG, thereby improving convergence stability.
  • SumTree-based Prioritized Experience Replay: To further accelerate learning, AQMDRL employs a SumTree data structure for prioritized experience replay. This mechanism intelligently samples experiences from the replay buffer based on their "importance" (measured by TD error), ensuring that more informative experiences are learned from more frequently, thus boosting the convergence rate.

The system architecture itself comprises three planes:

  1. Data Plane: SDN-enabled switches that forward traffic and report network states.
  2. Control Plane: The SDN controller that manages flow tables and uses the application plane's output to route traffic.
  3. Application Plane: Hosts the AQMDRL agent, which receives network states, generates link weights (actions), and communicates them to the control plane.

The agent's objective is to maximize a reward function that balances minimizing link delay (higher link-delay coefficient) and minimizing load imbalance (lower load-balancing coefficient).

Datasets

The paper does not introduce new datasets. Instead, it utilizes a simulated network environment for training and evaluation. The experimental setup involves the Mininet emulation platform and the RYU controller, simulating a network topology based on the NSFNET backbone. This simulated network consists of 14 switches and 21 links, each with a bandwidth of 10 Mbps. The experiments are conducted under varying traffic intensities (25%, 50%, 75%, and 100%) to assess the algorithm's performance under different network load conditions.

Key Insights and Ideas

  • Multistep learning is crucial for DRL stability: The paper strongly suggests that moving beyond single-step updates in DRL is essential for overcoming the inherent instability and over/underestimation issues, particularly in complex environments like SDN.
  • Prioritized experience replay significantly speeds up convergence: The use of SumTree for prioritized sampling demonstrates a practical way to make DRL algorithms learn more efficiently by focusing on the most informative data.
  • Traffic intensity dictates optimal DRL parameters: The study highlights that hyperparameters like the actor learning rate need to be dynamically adjusted based on traffic load to achieve optimal performance. High traffic requires a lower learning rate for stability, while low traffic benefits from a higher rate for faster exploration.
  • AQMDRL offers a robust trade-off: The proposed algorithm successfully balances the need for fast convergence with stable performance, outperforming existing DRL methods in key QoS metrics.

Key Results

  • Faster Convergence: AQMDRL exhibits the fastest convergence rate among the compared DRL algorithms (DDPG, TD3) across all traffic intensity scenarios.
  • Improved Stability: The reward curves for AQMDRL are significantly more stable, showing fewer fluctuations, especially after convergence.
  • Reduced Network Latency: AQMDRL achieves the highest average link-delay coefficient, indicating lower network transmission delays. For instance, at 100% traffic intensity, it shows a 5% higher link-delay coefficient than DDPG.
  • Enhanced Load Balancing: AQMDRL consistently achieves the lowest average load-balancing coefficient, signifying better network link utilization. At 50% traffic intensity, it is 16% lower than TD3 and 27% lower than DDPG.
  • Higher Throughput: AQMDRL demonstrates superior throughput, especially under high traffic loads, where it is over 5% higher than other algorithms due to its improved convergence and stability.
  • Computational Cost: While AQMDRL has a higher calculation cost than DDPG due to SumTree, it is lower than TD3, and its superior performance and convergence rate justify this trade-off.

Limitations and Future Work

  • Limitations:
    • The study primarily focuses on a simulated environment. Real-world network deployment might introduce additional complexities.
  • While AQMDRL improves upon DDPG and TD3, the paper acknowledges that when traffic changes drastically, the agent still needs to re-explore, leading to occasional reward jumps.
  • Future Work:
    • The authors propose enhancing the DRL neural network to better capture network state characteristics.
  • Exploring more advanced techniques like multi-agent reinforcement learning, multi-objective reinforcement learning, and multi-strategy RL is suggested to tackle larger-scale networks and optimize QoS more effectively.

Summary updated

Abstract

Software-defined networking (SDN) has become one of the critical technologies for data center networks, as it can improve network performance from a global perspective using artificial intelligence algorithms. Due to the strong decision-making and generalization ability, deep reinforcement learning (DRL) has been used in SDN intelligent routing and scheduling mechanisms. However, traditional deep reinforcement learning algorithms present the problems of slow convergence rate and instability, resulting in poor network quality of service (QoS) for an extended period before convergence. Aiming at the above problems, we propose an automatic QoS architecture based on multistep DRL (AQMDRL) to optimize the QoS performance of SDN. AQMDRL uses a multistep approach to solve the overestimation and underestimation problems of the deep deterministic policy gradient (DDPG) algorithm. The multistep approach uses the maximum value of the n-step action currently estimated by the neural network instead of the one-step Q-value function, as it reduces the possibility of positive error generated by the Q-value function and can effectively improve convergence stability. In addition, we adapt a prioritized experience sampling based on SumTree binary trees to improve the convergence rate of the multistep DDPG algorithm. Our experiments show that the AQMDRL we proposed significantly improves the convergence performance and effectively reduces the network transmission delay of SDN over existing DRL algorithms.
BibTeX
@inproceedings{chen2022aqmdrl,
  author = {Junyan Chen and Cenhuishan Liao and Yong Wang and Lei Jin and Xiaoye Lu and Xiaolan Xie and Rui Yao},
  title = {{AQMDRL: Automatic Quality of Service Architecture Based on Multistep Deep Reinforcement Learning in Software-Defined Networking}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s23010429},
}
Figure