Two Tier Slicing Resource Allocation Algorithm Based on Deep Reinforcement Learning and Joint Bidding in Wireless Access Networks

Geng Chen, Xu Zhang, Fei Shen, Qingtian ZengSensors 2022 · DOI

TL;DR

This paper introduces a novel two-tier resource allocation algorithm for 5G network slicing that combines Deep Reinforcement Learning (DRL) with a joint bidding strategy. By treating Mobile Virtual Network Operators (MVNOs) as agents, the algorithm optimizes resource utilization for both infrastructure providers (InPs) and end-users, leading to improved system utility and user quality of experience.

What Problem Does This Paper Address?

The increasing demand for diverse mobile services in the 5G era necessitates efficient resource allocation within wireless networks. Network slicing (NS) allows for the creation of virtual networks tailored to specific service requirements (e.g., eMBB, mMTC, URLLC). However, effectively allocating limited base station (BS) resources among these slices and the diverse needs of individual users remains a significant challenge. Traditional methods often struggle with the dynamic nature of user demands and channel conditions, leading to suboptimal resource utilization and potentially poor user quality of experience (QoE). This paper aims to address this by proposing a more intelligent and adaptive resource allocation framework.

Methods

The core of this paper's contribution lies in its two-tier resource allocation approach, leveraging Deep Reinforcement Learning (DRL) and a joint bidding mechanism.

  • Two-Tier Architecture: The system is divided into two levels:
  • Upper Tier: This level focuses on resource allocation between the infrastructure provider (InP) and Mobile Virtual Network Operators (MVNOs). MVNOs act as agents, submitting bids for network resources. The InP then allocates resources proportionally to these bids. This tier utilizes a combination of bidding and the Deep Q-Network (DQN) algorithm to optimize MVNO resource acquisition.
  • Lower Tier: Within each MVNO, resources are allocated to individual mobile users (MUs). This tier employs the Dueling Deep Q-Network (Dueling DQN) algorithm for iterative learning to find optimal resource distribution strategies that maximize user service quality and MVNO utility.
  • DRL Algorithms:
    • Deep Q-Network (DQN): Used in the upper tier to enable MVNOs to learn optimal bidding strategies to acquire resources from the InP.
  • Dueling Deep Q-Network (Dueling DQN): Utilized in the lower tier for MVNOs to distribute their acquired resources to users, optimizing for user satisfaction and overall MVNO utility. Dueling DQN is chosen for its ability to better separate state value and advantage functions, potentially leading to more efficient learning.
  • Joint Bidding: This strategy is integrated into the upper tier, allowing MVNOs to strategically bid for resources, considering both their own needs and the potential for overbidding. A penalty function is also incorporated to prevent excessive bidding.

Datasets

Not explicitly discussed in the paper. The authors mention using a simulation environment built on a Python platform. The simulation parameters are detailed, including system bandwidth, resource block specifications, transmission power, signal path, number of MVNOs, number of users, user package size, and SLA speeds for different service types (VoLTE, eMBB, URLLC).

Key Insights and Ideas

  • MVNOs as Strategic Bidders: Introducing MVNOs as independent agents that bid for resources adds a layer of strategic interaction, mimicking real-world market dynamics and potentially leading to more efficient resource allocation than a centralized approach.
  • Two-Tier Optimization for Holistic Performance: Decomposing the problem into upper (InP-MVNO) and lower (MVNO-User) tiers allows for simultaneous optimization of both infrastructure utilization and end-user satisfaction, addressing a critical trade-off.
  • Dueling DQN's Advantage in User-Centric Allocation: The paper suggests that Dueling DQN's architecture is particularly well-suited for the lower-tier allocation problem, where fine-grained optimization of user service quality is paramount.
  • Bidding as a Mechanism for Resource Discovery: The bidding mechanism in the upper tier helps the InP gauge demand and allocate resources more effectively, especially when direct user information is not available to the InP.

Key Results

The proposed DRL-based joint bidding algorithm demonstrates significant improvements over existing methods:

  • Upper Tier Performance:
    • The proposed algorithm achieved approximately 5.4% higher total system utility and revenue compared to Double DQN.
  • It also showed a 2.6% improvement over Dueling DQN in the upper tier.
  • Lower Tier Performance:
    • User service quality (QoE) was more stable.
  • System utility and SE (Service Efficiency) were 0.5-2.7% higher than DQN and Double DQN.
  • The algorithm exhibited faster convergence.
  • Qualitative Findings: The Dueling DQN algorithm, when applied to the lower tier, resulted in more stable and less volatile QoE curves compared to other methods. The SE and utility curves showed significant optimization with increasing iterations, indicating the effectiveness of the DRL approach.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Static User Location and Demand: The current model assumes static user locations and service demands. Future work could incorporate dynamic changes in user location and service demand to better reflect real-world scenarios.
  • Single BS Scenario: The study focuses on a single base station. Extending the model to a multi-BS environment would be a valuable next step.
  • Advanced DL Algorithms: The paper suggests exploring more advanced Deep Learning algorithms to further enhance the bidding and allocation schemes.
  • Real-World Deployment: While simulations show promising results, real-world deployment and testing would be crucial to validate the algorithm's performance under diverse network conditions.

Summary updated

Abstract

Network slicing (NS) is an emerging technology in recent years, which enables network operators to slice network resources (e.g., bandwidth, power, spectrum, etc.) in different types of slices, so that it can adapt to different application scenarios of 5 g network: enhanced mobile broadband (eMBB), massive machine-type communications (mMTC) and ultra-reliable and low-latency communications (URLLC). In order to allocate these sliced network resources more effectively to users with different needs, it is important that manage the allocation of network resources. Actually, in the practical network resource allocation problem, the resources of the base station (BS) are limited and the demand of each user for mobile services is different. To better deal with the resource allocation problem, more effective methods and algorithms have emerged in recent years, such as the bidding method, deep learning (DL) algorithm, ant colony algorithm (AG), and wolf colony algorithm (WPA). This paper proposes a two tier slicing resource allocation algorithm based on Deep Reinforcement Learning (DRL) and joint bidding in wireless access networks. The wireless virtual technology divides mobile operators into infrastructure providers (InPs) and mobile virtual network operators (MVNOs). This paper considers a single base station, multi-user shared aggregated bandwidth radio access network scenario and joins the MVNOs to fully utilize base station resources, and divides the resource allocation process into two tiers. The algorithm proposed in this paper takes into account both the utilization of base station (BS) resources and the service demand of mobile users (MUs). In the upper tier, each MVNO is treated as an agent and uses a combination of bidding and Deep Q network (DQN) allows the MVNO to get more resources from the base station. In the lower tier allocation process, each MVNO distributes the received resources to the users who are connected to it, which also uses the Dueling DQN method for iterative learning to find the optimal solution to the problem. The results show that in the upper tier, the total system utility function and revenue obtained by the proposed algorithm are about 5.4% higher than double DQN and about 2.6% higher than Dueling DQN; In the lower tier, the user service quality obtained by using the proposed algorithm is more stable, the system utility function and Se are about 0.5-2.7% higher than DQN and Double DQN, but the convergence is faster.
BibTeX
@inproceedings{chen2022two,
  author = {Geng Chen and Xu Zhang and Fei Shen and Qingtian Zeng},
  title = {{Two Tier Slicing Resource Allocation Algorithm Based on Deep Reinforcement Learning and Joint Bidding in Wireless Access Networks}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22093495},
}
Figure