Abstract
In ultra-dense wireless networks (UDNs), frequent handovers (HO) significantly degrade user experience through signaling overhead and data interruptions. This paper presents a novel two-layer framework utilizing Asynchronous Multi-User Deep Reinforcement Learning (DRL) to optimize HO control. By clustering users with similar mobility patterns and employing an asynchronous advantage actor-critic (A3C) architecture with Long Short-Term Memory (LSTM) units, the proposed system effectively reduces unnecessary handovers while maintaining high throughput. Experimental results demonstrate that this approach outperforms traditional 3GPP schemes and standard machine learning baselines, offering a scalable solution for heterogeneous mobility environments.
Introduction
The evolution toward 5G and beyond has led to the deployment of ultra-dense networks (UDNs), where small base stations (SBSs) are packed tightly to maximize capacity. However, this density introduces a "frequent handover" problem. As User Equipments (UEs) move, they may constantly switch between base stations, leading to increased signaling overhead and brief interruptions in data flow.
Traditional HO control relies on static parameters such as Hysteresis Margins (HHM) and Time-to-Trigger (TTT). These fixed rules are often suboptimal in environments with heterogeneous mobility—where a high-speed vehicle and a walking pedestrian require vastly different HO strategies. To address this, researchers are turning to Deep Reinforcement Learning (DRL) to create adaptive controllers capable of learning optimal policies from environmental interactions.
The Proposed Framework
The authors propose a "divide and conquer" strategy to manage the complexity of multi-user HO control in UDNs through a two-layer architecture.
Layer 1: Centralized Clustering
To ensure scalability, a central controller first partitions UEs into clusters using a K-means algorithm based on mobility features like coordinates and velocity. This clustering assumes that users with similar movement patterns can benefit from a shared HO policy.
As shown in the architecture above, the central controller manages multiple clusters, each served by a dedicated parameter server. This hierarchy allows the system to handle large-scale deployments by localizing the learning process within specific geographic or mobility-based groups.
Layer 2: Asynchronous Multi-User DRL
Within each cluster, the system implements the Asynchronous Advantage Actor-Critic (A3C) framework. In this setup, individual UEs act as "workers" that interact with their local environment.
Each UE maintains its own local copy of the policy and value functions. As UEs move and experience different signal conditions, they calculate gradients based on their rewards—a weighted sum of throughput and HO penalties—and push these updates to the cluster's parameter server. The server then updates the global model and sends the new parameters back to the UEs. This asynchronous approach allows for parallel learning, which speeds up convergence and improves robustness.
Model Architecture and Training
The HO controller utilizes a Deep Neural Network (DNN) integrated with Long Short-Term Memory (LSTM) units. LSTM is vital for HO control because it enables the model to remember historical signal strengths and movement trends, allowing for more informed predictions of future connectivity needs.
To mitigate the "cold start" problem common in RL—where agents perform poorly during initial exploration—the authors employ a hybrid training approach. The DNN is first initialized using Supervised Learning (SL) on data generated by traditional 3GPP HO schemes. This "jump start" ensures that the agent begins with a functional baseline policy before fine-tuning it through RL exploration.
Key Results and Insights
The proposed framework was evaluated in a simulated UDN environment featuring multiple small base stations and realistic pathloss models. Key findings include:
- HO Rate Reduction: The A3C-based framework achieved significantly lower HO rates compared to traditional 3GPP protocols and the Upper Confidence Bound (UCB) algorithm.
- Throughput-HO Trade-off: By adjusting the weight factor ($\beta$) in the reward function, the system can navigate a Pareto frontier, allowing operators to prioritize either maximum throughput or minimum signaling overhead.
- Scalability and Convergence: The clustering mechanism proved essential for performance. By sharing parameters among similar users, the system reached higher state-value estimations much faster than non-clustered alternatives.
Conclusion and Future Work
This research demonstrates that combining mobility-based clustering with asynchronous DRL provides a powerful solution for handover management in dense networks. By leveraging the collective experience of multiple users and incorporating temporal memory via LSTMs, the framework achieves a balance between high data rates and low signaling overhead.
Future work will focus on addressing inter-cell interference and managing the "delayed gradient" problem inherent in asynchronous updates to further enhance system stability in highly dynamic environments.
Summary updated