Load Balancing for Ultradense Networks: A Deep Reinforcement Learning-Based Approach

Yue Xu, Wenjun Xu, Zhi Wang, Jiaru Lin, Shuguang Cui

TL;DR

This paper tackles the challenge of efficiently distributing mobile traffic across a dense network of small cell base stations (UDNs) using a novel two-layer deep reinforcement learning (DRL) approach. By intelligently clustering cells and employing an adaptive DRL algorithm with a safeguard mechanism, the authors achieve significantly improved load balancing performance and system stability.

What Problem Does This Paper Address?

The advent of 5G networks promises a massive increase in mobile data traffic, leading to the widespread deployment of Ultra-Dense Networks (UDNs). UDNs, characterized by a high density of small cell base stations (SBSs), offer improved link quality and network capacity. However, this densification exacerbates traffic fluctuations, leading to Quality of Service (QoS) degradation. Traditional load balancing methods struggle with the complexity, irregular topology, and dynamic user mobility inherent in UDNs. Existing reinforcement learning (RL) approaches are often limited by the curse of dimensionality when dealing with large state and action spaces. This paper aims to address these limitations by proposing a scalable, adaptive, and stable load balancing solution for UDNs.

Methods

The paper introduces a two-layer architecture for mobility load balancing (MLB):

  • Top Layer (Centralized Clustering): This layer dynamically groups SBSs into self-organized clusters based on historical load levels. It uses a tailored k-means algorithm, initializing centroids with the most overloaded SBSs and then grouping nearby SBSs based on geographical distance. This approach breaks down the large-scale problem into smaller, manageable sub-problems.
  • Bottom Layer (Self-Organized DRL-based MLB): This layer focuses on balancing the load within each cluster. It employs an off-policy DRL-based MLB algorithm that uses deep neural networks to approximate the optimal MLB policy. Key features include:
  • Asynchronous Parallel Learning: The algorithm can be trained in parallel, improving efficiency.
  • Joint Exploration with Multiple Behavior Policies: This allows traditional MLB methods to guide the DRL learning process, enhancing efficiency and stability, especially in the early stages.
  • Offline-Evaluation Based Safeguard Mechanism: This mechanism runs an online control branch and an offline policy learning branch in parallel. The online branch always uses the best-trained policy from the offline branch, ensuring stable online performance and enabling safe exploration of new policies.

Datasets

Not discussed in the paper. The authors rely on simulations to evaluate their proposed approach.

Key Insights and Ideas

  • Hierarchical Load Balancing: Decomposing the large-scale UDN load balancing problem into smaller, cluster-based sub-problems is crucial for scalability and efficient management.
  • DRL for Dynamic Adaptation: Deep Reinforcement Learning, with its ability to learn from experience and adapt to unknown environments, is a powerful tool for handling the dynamic and complex nature of UDN traffic.
  • Guided Exploration: Using existing, albeit suboptimal, MLB methods to guide the DRL exploration process significantly improves learning efficiency and stability, mitigating the risks of purely random exploration.
  • Safeguarding Online Performance: The offline-evaluation safeguard mechanism is critical for real-world deployment, ensuring that the system operates with a stable and well-trained policy, preventing performance degradation during the learning phase.

Key Results

The proposed DRL-based approach demonstrated superior performance compared to baseline methods (static/adaptive rule-based, Q-learning, and no MLB) across various metrics:

  • Load Balancing Performance: The DRL-based algorithm achieved significantly lower handover failure rates (HFR) and load standard deviations, indicating more balanced traffic distribution. For instance, under a CBR of 112 kbps, the DRL-based algorithms kept the maximum SBS load below 55%, outperforming rule-based methods (below 67%) and no MLB (fluctuating around 74%).
  • Scalability: The two-layer architecture showed superior scalability compared to fully centralized approaches as the number of SBSs increased.
  • Learning Efficiency: The DRL approach with multiple behavior policies converged remarkably faster than with a single behavior policy.
  • System Stability: The safeguard mechanism ensured stable and non-decreasing MLB performance over time, even during re-clustering events.

Limitations and Future Work

  • Feature Selection: The paper acknowledges that the chosen state features (SBS load and edge user fraction) are not unique and other combinations could be explored for improved performance.
  • Clustering Algorithm: The proposed k-means clustering is one possible solution; future work could explore other dynamic clustering methods considering user mobility patterns, social relationships, or radio propagation.
  • Convergence Guarantee: While empirical results show convergence, the use of deep neural networks means theoretical convergence is not guaranteed, which is a common challenge in DRL.
  • Generalizability: The paper suggests that the proposed architecture, algorithm, and mechanism are promising for other large-scale network control problems by simply changing the learning context. This implies potential for broader applications in areas like IoT access control.

Summary updated

Abstract

In this paper, we propose a deep reinforcement learning (DRL) based mobility load balancing (MLB) algorithm along with a two-layer architecture to solve the large-scale load balancing problem for ultra-dense networks (UDNs). Our contribution is three-fold. First, this work proposes a two-layer architecture to solve the large-scale load balancing problem in a self-organized manner. The proposed architecture can alleviate the global traffic variations by dynamically grouping small cells into self-organized clusters according to their historical loads, and further adapt to local traffic variations through intra-cluster load balancing afterwards. Second, for the intra-cluster load balancing, this paper proposes an off-policy DRL-based MLB algorithm to autonomously learn the optimal MLB policy under an asynchronous parallel learning framework, without any prior knowledge assumed over the underlying UDN environments. Moreover, the algorithm enables joint exploration with multiple behavior policies, such that the traditional MLB methods can be used to guide the learning process thereby improving the learning efficiency and stability. Third, this work proposes an offline-evaluation based safeguard mechanism to ensure that the online system can always operate with the optimal and well-trained MLB policy, which not only stabilizes the online performance but also enables the exploration beyond current policies to make full use of machine learning in a safe way. Empirical results verify that the proposed framework outperforms the existing MLB methods in general UDN environments featured with irregular network topologies, coupled interferences, and random user movements, in terms of the load balancing performance.
BibTeX
@article{xu2019load,
  author = {Yue Xu and Wenjun Xu and Zhi Wang and Jiaru Lin and Shuguang Cui},
  title = {{Load Balancing for Ultradense Networks: A Deep Reinforcement Learning-Based Approach}},
  journal = {IEEE Internet of Things Journal},
  year = {2019},
  url = {https://openreview.net/forum?id=2Bgwfh5JPG},
}
Figure