Slicing Resource Allocation Based on Dueling DQN for eMBB and URLLC Hybrid Services in Heterogeneous Integrated Networks

Geng Chen, Rui Shao, Fei Shen, Qingtian ZengSensors 2023 · DOI

TL;DR

This paper proposes a Dueling Deep Q-Network (DQN) based approach for dynamic resource allocation and scheduling in 5G networks, specifically for hybrid Enhanced Mobile Broadband (eMBB) and Ultra-Reliable Low-Latency Communication (URLLC) services. By prioritizing URLLC's strict latency requirements and employing a novel resource scheduling mechanism, the Dueling DQN algorithm significantly improves network utility and stability compared to traditional RL methods.

What Problem Does This Paper Address?

Modern 5G networks need to cater to a diverse range of services with vastly different requirements. Two prominent examples are eMBB, which demands high bandwidth for data-intensive applications like video streaming, and URLLC, which requires extremely low latency and high reliability for critical applications like remote surgery or autonomous driving. Network slicing is a key technology for managing these diverse needs by creating virtual networks tailored to specific service requirements. However, efficiently allocating and scheduling limited network resources (like bandwidth) between these competing services, especially when URLLC demands spike, is a significant challenge. Existing methods often struggle to balance the conflicting QoS requirements, sometimes sacrificing eMBB quality for URLLC's strict latency, or vice-versa. This paper tackles the problem of optimizing resource allocation and scheduling for hybrid eMBB and URLLC services in a heterogeneous network environment to maximize overall network utility while ensuring service quality.

Methods

The core of this paper is a novel resource allocation and scheduling algorithm based on Dueling Deep Q-Network (Dueling DQN). Here's a breakdown of the key techniques:

  • Dueling DQN: This is an advancement over standard DQN. It separates the value function into a state-value function and an advantage function. This architecture is particularly effective for problems where the impact of actions on the overall state value is crucial, allowing for more efficient learning and better handling of complex state spaces.
  • Resource Scheduling Mechanism: To prioritize URLLC's stringent requirements, the paper introduces a mechanism where, under high demand or insufficient bandwidth, a portion of eMBB bandwidth can be dynamically reallocated to URLLC users. This is done carefully to minimize the impact on eMBB's Quality of Experience (QoE).
  • Reward Clipping: To enhance training stability and address the different scales of QoE and Spectrum Efficiency (SE), a reward clipping mechanism is employed. This helps the agent learn more effectively by providing more consistent and informative reward signals.
  • Bandwidth Allocation Resolution (b): The paper investigates the impact of the granularity of bandwidth allocation. Choosing an appropriate resolution (e.g., 0.1 MHz) is shown to balance flexibility in resource allocation with the complexity of the action space.
  • ϵ-greedy Strategy: This standard reinforcement learning exploration strategy is used to balance exploiting known good actions with exploring new ones to discover potentially better solutions.

Datasets

The paper does not introduce new datasets. Instead, it uses a simulated environment to evaluate the proposed algorithm. The simulation parameters, including user distribution, packet standards, and channel characteristics (Rayleigh fading), are based on established 5G standards and references (specifically citing [22] for user and packet standards).

Key Insights and Ideas

  • Dueling DQN's Superiority: The Dueling DQN architecture demonstrably outperforms traditional Q-learning, DQN, and Double DQN in terms of stability and achieving higher network utility, especially in dynamic and complex scenarios like hybrid service slicing.
  • Strategic Resource Reallocation: The proposed resource scheduling mechanism is crucial. It allows for the dynamic reallocation of eMBB bandwidth to URLLC when needed, significantly improving URLLC's QoE without drastically degrading eMBB's performance. This highlights the importance of intelligent, context-aware resource management.
  • The "Sweet Spot" for Bandwidth Resolution: The choice of bandwidth allocation resolution (b) is not trivial. A too-fine resolution can lead to an overly complex action space, while a too-coarse resolution limits flexibility. The paper identifies 0.1 MHz as an optimal balance for their scenario.
  • Reward Shaping for Stability: The reward clipping mechanism is vital for training stability, especially when dealing with metrics of different magnitudes (like QoE and SE). It helps the agent learn to optimize for both simultaneously.

Key Results

  • Improved Network Utility: The Dueling DQN algorithm achieved an 11% improvement in network utility compared to Q-learning, 8% over DQN, and 2% over Double DQN.
  • Enhanced QoE and SE: The resource scheduling mechanism further improved URLLC's QoE by 3%, and overall SE and network utility by 4.5% and 7%, respectively.
  • High QoE Stability: The Dueling DQN algorithm consistently achieved near-optimal QoE (close to 1.0) for both eMBB and URLLC services after an initial training period, demonstrating remarkable stability.
  • Faster Convergence: Dueling DQN showed a more stable and faster convergence trend compared to the other algorithms tested.

Limitations and Future Work

  • Simulation-Based: The results are based on simulations, and real-world deployment might introduce additional complexities and performance variations.
  • Specific Scenario: The study focuses on a specific scenario with eMBB and URLLC services. Extending this to include other service types like mMTC (massive Machine Type Communications) would be a valuable future direction.
  • Parameter Sensitivity: While the paper explores the impact of bandwidth resolution (b), further in-depth analysis of the sensitivity to other hyperparameters (learning rate, discount factor, etc.) could be beneficial.
  • Future Work Suggestions: The authors implicitly suggest future work by highlighting the benefits of their approach. Expanding to more complex network topologies, incorporating user mobility more explicitly, and exploring other advanced DRL techniques could be promising avenues.

Summary updated

Abstract

In 5G/B5G communication systems, network slicing is utilized to tackle the problem of the allocation of network resources for diverse services with changing demands. We proposed an algorithm that prioritizes the characteristic requirements of two different services and tackles the problem of allocation and scheduling of resources in the hybrid services system with eMBB and URLLC. Firstly, the resource allocation and scheduling are modeled, subject to the rate and delay constraints of both services. Secondly, the purpose of adopting a dueling deep Q network (Dueling DQN) is to approach the formulated non-convex optimization problem innovatively, in which a resource scheduling mechanism and the ϵ-greedy strategy were utilized to select the optimal resource allocation action. Moreover, the reward-clipping mechanism is introduced to enhance the training stability of Dueling DQN. Meanwhile, we choose a suitable bandwidth allocation resolution to increase flexibility in resource allocation. Finally, the simulations indicate that the proposed Dueling DQN algorithm has excellent performance in terms of quality of experience (QoE), spectrum efficiency (SE) and network utility, and the scheduling mechanism makes the performance much more stable. In contrast with Q-learning, DQN as well as Double DQN, the proposed algorithm based on Dueling DQN improves the network utility by 11%, 8% and 2%, respectively.
BibTeX
@inproceedings{chen2023slicing,
  author = {Geng Chen and Rui Shao and Fei Shen and Qingtian Zeng},
  title = {{Slicing Resource Allocation Based on Dueling DQN for eMBB and URLLC Hybrid Services in Heterogeneous Integrated Networks}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23052518},
}
Figure