Intelligent Handover Algorithm for Vehicle-to-Network Communications With Double-Deep Q-Learning

Kang Tan, Duncan Bremner, Julien Le Kernec, Yusuf A. Sambo, Lei Zhang, Muhammad Ali ImranIEEE Trans. Vehicular Technology 2022 · DOI

Abstract

In high-mobility Vehicle-to-Network (V2N) environments, maintaining seamless connectivity is a significant challenge due to the limitations of traditional handover (HO) algorithms. This paper proposes a Double-Deep Q-Learning (DDQN) approach to optimize handover decisions. By leveraging standard cellular measurement reports, the DDQN agent learns to proactively trigger handovers, balancing signal quality against switching costs. Simulation results in a realistic urban environment demonstrate a 25.72% reduction in packet loss and an 11.56-second improvement in handover timing compared to the industry-standard A3 RSRP algorithm.

The Handover Challenge in Vehicular Networks

Vehicle-to-Network (V2N) communications are essential for modern intelligent transportation systems, supporting everything from real-time traffic updates to safety-critical autonomous driving data. However, vehicles moving at high speeds frequently transition between the coverage areas of different base stations (BSs). Current 5G and LTE standards utilize "hard handovers," where the connection to the serving BS is severed before a new connection is established.

To prevent "ping-pong" effects—where a device rapidly switches back and forth between two stations due to signal fluctuations—the industry-standard A3 RSRP algorithm employs fixed parameters: hysteresis ($\Delta$) and time-to-trigger ($\beta$). While effective in static scenarios, these fixed parameters often cause "late handovers" in high-speed urban environments, leading to significant signal degradation, packet loss, and service interruptions.

Standard A3 RSRP handover mechanism illustrating the delay introduced by hysteresis (Δ) and time-to-trigger (β).

Proposed Double-Deep Q-Learning Framework

To address these inefficiencies, this research introduces a Reinforcement Learning (RL) approach using Double-Deep Q-Learning (DDQN). Unlike standard Deep Q-Networks (DQN), which can suffer from overestimation bias regarding action values, DDQN provides more stable and accurate learning by decoupling action selection from evaluation.

System Architecture and Deployment

The proposed system is designed for practical deployment as a software update to a centralized or local base station controller. This eliminates the need for hardware modifications in vehicles (User Equipment, or UE). The interaction sequence involves the UE sending standard measurement reports to the serving BS, which are then forwarded to the Centralized RL Agent for inference.

Sequence diagram of the centralized RL agent-based handover process using DDQN.

RL Formulation

  • State Space ($S$): The agent observes the Reference Signal Received Power (RSRP) from all detectable base stations and identifies the current serving BS. These inputs are derived directly from standard cellular measurement reports.
  • Action Space ($A$): The agent determines the optimal target BS for the vehicle to connect to at any given time step.
  • Reward Function ($R$): The reward is designed to maximize signal strength while minimizing unnecessary handovers. The agent receives a positive reward for high RSRP and a negative penalty for each handover event, forcing the model to learn the optimal trade-off between connection quality and stability.

Experimental Setup

The algorithm was evaluated using the ns-3 network simulator and the LENA LTE module. To ensure real-world relevance, the researchers modeled a 2x2 km section of Glasgow city center, U.K.

Vehicle trajectories were generated via the Google Maps Directions API to reflect actual traffic patterns and speeds. Furthermore, the simulation incorporated trace-based fading generated in MATLAB to account for the Doppler effect and signal obstructions common in dense urban "canyons." The model was trained on 18 major routes and validated on modified trajectories to test its generalization capabilities.

Performance Evaluation

The DDQN algorithm significantly outperformed the A3 RSRP baseline across several critical metrics:

  1. Packet Loss Reduction: The algorithm achieved an average reduction in packet loss of 25.72% per handover. Over the duration of the full simulation, the total accumulated packet loss was reduced by 42.62%.
  2. Handover Timing: The DDQN agent learned to trigger handovers an average of 11.56 seconds earlier than the baseline, effectively eliminating the "late handover" problem without increasing the total number of handovers.
  3. Signal Quality: During the critical moments of transition between base stations, the DDQN approach maintained a Signal-to-Interference-plus-Noise Ratio (SINR) that was up to 12.5 dB higher than the standard method.

Conclusion

By utilizing standard RSRP measurements, the DDQN-based handover algorithm offers a high-performance, deployable solution for vehicular networks. It moves beyond reactive, parameter-fixed triggers to a proactive, geography-aware intelligence that significantly improves the reliability of V2N services. Future work will focus on extending this framework to multi-Radio Access Technology (multi-RAT) environments, such as seamless switching between 5G and Wi-Fi, and validating the model within the emerging 5G-NR (New Radio) standards.

Summary updated

BibTeX
@article{tan2022intelligent,
  author = {Kang Tan and Duncan Bremner and Julien Le Kernec and Yusuf A. Sambo and Lei Zhang and Muhammad Ali Imran},
  title = {{Intelligent Handover Algorithm for Vehicle-to-Network Communications With Double-Deep Q-Learning}},
  journal = {IEEE Trans. Vehicular Technology},
  year = {2022},
  doi = {10.1109/tvt.2022.3169804},
}
Figure