Queue-Learning: A Reinforcement Learning Approach for Providing Quality of Service

Majid Raeis, Ali Tizghadam, Alberto Leon-GarciaAAAI 2021 · DOI

Introduction

In the era of cloud computing and Network Function Virtualization (NFV), services are typically structured as "service chains"—sequences of processing nodes that packets must traverse. Maintaining strict Quality of Service (QoS), such as guaranteeing that 90% of packets experience an end-to-end delay of less than 10ms, is a critical challenge. Traditional queueing theory often struggles with the complex, bursty traffic patterns of real-world networks, while standard Reinforcement Learning (RL) lacks the mathematical framework to provide explicit probabilistic guarantees.

"Queue-Learning" addresses this by integrating queueing theory models with Deep Reinforcement Learning to dynamically manage service rates while satisfying specific QoS constraints.

The Queue-Learning Framework

The system models a service chain as a tandem network of $N$ multi-server queues. To manage this environment, the authors utilize the Deep Deterministic Policy Gradient (DDPG) algorithm, which is well-suited for the continuous action space of service rate adjustments.

The Queue-Learning architecture: A centralized service rate controller uses queue lengths ($s_t$) and rewards ($r_t$) to determine optimal service rates ($a_t$) for each node in the chain.

The controller observes the state $s_t$ (current queue lengths) and outputs an action $a_t$ (service rates $\mu_1, \dots, \mu_N$). The core innovation lies in the reward function, which is derived from the Lagrangian dual of a QoS optimization problem. This formulation allows the reward to balance two competing objectives:

  1. QoS Satisfaction: Penalizing instances where end-to-end delay ($d$) exceeds a defined upper bound ($d_{ub}$).
  2. Resource Efficiency: Minimizing the total service rate to prevent over-provisioning.

Experimental Evaluation

The framework was tested in a simulated environment using Gamma distributions for inter-arrival and service times, providing a more realistic test of "bursty" traffic than standard Poisson models. The setup involved a three-node Virtual Network Function (VNF) chain with varying numbers of servers at each stage.

Key Results and the $\lambda$ Trade-off

A critical component of the Queue-Learning approach is the hyper-parameter $\lambda$, which acts as a weight for the QoS constraint in the reward function. By tuning $\lambda$, operators can navigate the trade-off between strict delay bounds and resource costs.

Impact of the hyper-parameter $\lambda$ on (a) the probability of delay violation and (b) the total service rate (Sum-Rate). The point $\lambda^*$ indicates the optimal balance where QoS constraints are met with minimal resource usage.

As shown in the results, the controller successfully identifies the "sweet spot" ($\lambda^*$) where the violation probability falls just below the target threshold (e.g., 0.1), ensuring that QoS is maintained without wasting expensive computational resources. Furthermore, the study found that the length of the control interval ($T$) is vital; shorter time slots allow for more frequent interactions, leading to faster convergence and greater stability against traffic spikes.

Conclusion

Queue-Learning provides a robust, interpretable method for managing QoS in tandem service chains. By grounding the RL reward function in Lagrangian optimization, it transforms a "black box" learning process into a tool capable of providing explicit performance guarantees. Future work will focus on extending this centralized approach to more complex, non-linear network topologies and exploring decentralized multi-agent configurations.

Summary updated

BibTeX
@inproceedings{raeis2021queue,
  author = {Majid Raeis and Ali Tizghadam and Alberto Leon-Garcia},
  title = {{Queue-Learning: A Reinforcement Learning Approach for Providing Quality of Service}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i1.16123},
}
Figure