Handover Control in Wireless Systems via Asynchronous Multi-User Deep Reinforcement Learning

Zhi Wang, Yue Xu, Lihua Li, Hui Tian, Shuguang Cui

Abstract

In ultra-dense wireless networks (UDNs), frequent handovers (HO) significantly degrade user experience through signaling overhead and data interruptions. This paper presents a novel two-layer framework utilizing Asynchronous Multi-User Deep Reinforcement Learning (DRL) to optimize HO control. By clustering users with similar mobility patterns and employing an asynchronous advantage actor-critic (A3C) architecture with Long Short-Term Memory (LSTM) units, the proposed system effectively reduces unnecessary handovers while maintaining high throughput. Experimental results demonstrate that this approach outperforms traditional 3GPP schemes and standard machine learning baselines, offering a scalable solution for heterogeneous mobility environments.

Introduction

The evolution toward 5G and beyond has led to the deployment of ultra-dense networks (UDNs), where small base stations (SBSs) are packed tightly to maximize capacity. However, this density introduces a "frequent handover" problem. As User Equipments (UEs) move, they may constantly switch between base stations, leading to increased signaling overhead and brief interruptions in data flow.

Traditional HO control relies on static parameters such as Hysteresis Margins (HHM) and Time-to-Trigger (TTT). These fixed rules are often suboptimal in environments with heterogeneous mobility—where a high-speed vehicle and a walking pedestrian require vastly different HO strategies. To address this, researchers are turning to Deep Reinforcement Learning (DRL) to create adaptive controllers capable of learning optimal policies from environmental interactions.

The Proposed Framework

The authors propose a "divide and conquer" strategy to manage the complexity of multi-user HO control in UDNs through a two-layer architecture.

Layer 1: Centralized Clustering

To ensure scalability, a central controller first partitions UEs into clusters using a K-means algorithm based on mobility features like coordinates and velocity. This clustering assumes that users with similar movement patterns can benefit from a shared HO policy.

Hierarchical system architecture showing the central controller, clustered parameter servers, and localized areas.

As shown in the architecture above, the central controller manages multiple clusters, each served by a dedicated parameter server. This hierarchy allows the system to handle large-scale deployments by localizing the learning process within specific geographic or mobility-based groups.

Layer 2: Asynchronous Multi-User DRL

Within each cluster, the system implements the Asynchronous Advantage Actor-Critic (A3C) framework. In this setup, individual UEs act as "workers" that interact with their local environment.

The asynchronous training mechanism where multiple UEs (workers) interact with the environment and update a shared parameter server.

Each UE maintains its own local copy of the policy and value functions. As UEs move and experience different signal conditions, they calculate gradients based on their rewards—a weighted sum of throughput and HO penalties—and push these updates to the cluster's parameter server. The server then updates the global model and sends the new parameters back to the UEs. This asynchronous approach allows for parallel learning, which speeds up convergence and improves robustness.

Model Architecture and Training

The HO controller utilizes a Deep Neural Network (DNN) integrated with Long Short-Term Memory (LSTM) units. LSTM is vital for HO control because it enables the model to remember historical signal strengths and movement trends, allowing for more informed predictions of future connectivity needs.

To mitigate the "cold start" problem common in RL—where agents perform poorly during initial exploration—the authors employ a hybrid training approach. The DNN is first initialized using Supervised Learning (SL) on data generated by traditional 3GPP HO schemes. This "jump start" ensures that the agent begins with a functional baseline policy before fine-tuning it through RL exploration.

Key Results and Insights

The proposed framework was evaluated in a simulated UDN environment featuring multiple small base stations and realistic pathloss models. Key findings include:

  • HO Rate Reduction: The A3C-based framework achieved significantly lower HO rates compared to traditional 3GPP protocols and the Upper Confidence Bound (UCB) algorithm.
  • Throughput-HO Trade-off: By adjusting the weight factor ($\beta$) in the reward function, the system can navigate a Pareto frontier, allowing operators to prioritize either maximum throughput or minimum signaling overhead.
  • Scalability and Convergence: The clustering mechanism proved essential for performance. By sharing parameters among similar users, the system reached higher state-value estimations much faster than non-clustered alternatives.

Conclusion and Future Work

This research demonstrates that combining mobility-based clustering with asynchronous DRL provides a powerful solution for handover management in dense networks. By leveraging the collective experience of multiple users and incorporating temporal memory via LSTMs, the framework achieves a balance between high data rates and low signaling overhead.

Future work will focus on addressing inter-cell interference and managing the "delayed gradient" problem inherent in asynchronous updates to further enhance system stability in highly dynamic environments.

Summary updated

BibTeX
@article{wang2018handover,
  author = {Zhi Wang and Yue Xu and Lihua Li and Hui Tian and Shuguang Cui},
  title = {{Handover Control in Wireless Systems via Asynchronous Multi-User Deep Reinforcement Learning}},
  journal = {IEEE Internet of Things Journal},
  year = {2018},
  url = {https://openreview.net/forum?id=xZvwPD-5TSp},
}
Figure