Deep Reinforcement Learning-Based Network Slicing for Beyond 5G

Kyungjoo Suh, Sunwoo Kim, Yongjun Ahn, Seungnyun Kim, Hyungyu Ju, Byonghyo ShimIEEE Access 2022 · OpenReview

Abstract

In the Beyond 5G (B5G) era, network slicing has emerged as a critical technique for flexibly supporting diverse wireless services. By partitioning a single physical network into multiple independent slices, operators can cater to the distinct requirements of enhanced Mobile Broadband (eMBB), Ultra-Reliable Low-Latency Communications (URLLC), and massive Machine-Type Communications (mMTC). However, the combinatorial complexity of resource block (RB) allocation makes optimal slicing an NP-hard problem. This paper proposes a Deep Reinforcement Learning (DRL)-based network slicing technique (DRL-NS) that maximizes long-term throughput while strictly adhering to Quality of Service (QoS) constraints. The core innovation is an action elimination strategy that prunes the action space of undesirable decisions, significantly improving training efficiency and system performance.

The Challenge of Beyond 5G Network Slicing

B5G systems must simultaneously manage three service categories with conflicting goals:

  1. eMBB: Demands high data rates for applications like 4K streaming and AR/VR.
  2. URLLC: Requires near-instantaneous response times for autonomous vehicles and industrial automation.
  3. mMTC: Supports a massive density of low-power IoT sensors.
Overview of 5G network slicing and resource allocation for different use cases.

Traditional analytical methods struggle with the dynamic, time-varying nature of wireless channels. While standard DRL offers a potential solution, it often suffers from an astronomical action space. In a typical setup, a DRL agent might waste most of its training time exploring "illegal" resource allocations that violate latency or rate requirements, leading to slow convergence and suboptimal policies.

Proposed DRL-NS Framework

The proposed DRL-NS framework utilizes a Deep Q-Network (DQN) where the base station (gNB) serves as the intelligent agent. To overcome the exploration inefficiency of standard DRL, the authors introduce Action Space Reduction via Action Elimination.

Before the agent selects an action, the system applies two sequential filters:

  • URLLC Test: Eliminates any RB allocation that cannot meet the strict maximum packet delay requirements for URLLC users.
  • eMBB Test: Removes allocations that fail to satisfy the minimum data rate thresholds for eMBB users.

By narrowing the search to a "desirable action space," the agent only learns from feasible actions. The state space incorporates current and previous channel matrices to capture temporal correlations, while the reward function is tied to total system throughput, incentivizing efficiency within the "safe" filtered space.

Experimental Setup

The framework was evaluated in a high-fidelity simulation environment:

  • Channel Modeling: Rayleigh fading for small-scale effects and the Hata-COST231 model for large-scale path loss.
  • Network Topology: 10 small cells (gNBs) serving 4 User Equipments (UEs) with a 200m inter-site distance.
  • Implementation: Developed using TensorFlow and trained on an NVIDIA GeForce Titan Xp.

Results and Performance Analysis

Numerical results demonstrate that DRL-NS significantly outperforms traditional heuristics and vanilla DRL.

Comparison of cumulative reward during training for DRL-NS vs. standard DQN.

Key findings include:

  • Throughput Gains: DRL-NS achieved approximately 25% higher throughput than equal allocation and 15% higher than regression-tree-based methods.
  • Training Efficiency: As shown in the cumulative reward plots, the action elimination strategy allows the agent to converge much faster than a standard DQN. DRL-NS outperformed vanilla DQN by 10% in final throughput, proving that the filtered search space leads to a superior policy.
  • Robustness: The system maintained high performance even under stringent QoS requirements where heuristic methods typically fail.

Conclusion and Future Work

DRL-NS provides a robust solution for the complex resource allocation problems inherent in B5G network slicing. By integrating domain-specific constraints directly into the RL exploration process, the framework achieves high sample efficiency and superior throughput.

Future research directions include:

  • Reducing Computational Overhead: Parallelizing the action filtering process to scale to larger numbers of RBs.
  • Multi-Agent DRL (MADRL): Transitioning to a decentralized model where multiple gNBs coordinate autonomously.
  • Uplink Integration: Extending the framework to handle uplink constraints, such as user device power limitations.

Summary updated

Abstract

With the advent of 5G era, network slicing has received a great deal of attention as a means to support a variety of wireless services in a flexible manner. Network slicing is a technique to divide a single physical resource network into multiple slices supporting independent services. In beyond 5G (B5G) systems, the main goal of network slicing is to assign the physical resource blocks (RBs) such that the quality of service (QoS) requirements of eMBB, URLLC, and mMTC services are satisfied. Since the goal of each service category is dearly distinct and the computational burden caused by the increased number of time slots is huge, it is in general very difficult to assign RB properly. In this paper, we propose a deep reinforcement learning (DRL)-based network slicing technique to find out the resource allocation policy maximizing the long-term throughput while satisfying the QoS requirements in the B5G systems. Key ingredient of the proposed technique is to reduce the action space by eliminating undesirable actions that cannot satisfy the QoS requirements. Numerical results demonstrate that the proposed technique is effective in maximizing the long-term throughput and handling the coexistence of use cases in the B5G environments.
BibTeX
@inproceedings{suh2022deep,
  author = {Kyungjoo Suh and Sunwoo Kim and Yongjun Ahn and Seungnyun Kim and Hyungyu Ju and Byonghyo Shim},
  title = {{Deep Reinforcement Learning-Based Network Slicing for Beyond 5G}},
  booktitle = {IEEE Access},
  year = {2022},
  url = {https://openreview.net/forum?id=eqjg508X9B},
}
Figure