Abstract
Unmanned aerial base stations (UABSs) can be deployed in vehicular wireless networks to support applications such as extended sensing via vehicle-to-everything (V2X) services. A key problem in such systems is designing algorithms that can efficiently optimize the trajectory of the UABS in order to maximize coverage. In existing solutions, such optimization is carried out from scratch for any new traffic configuration, often by means of conventional reinforcement learning (RL). In this paper, we propose the use of continual meta-RL as a means to transfer information from previously experienced traffic configurations to new conditions, with the goal of reducing the time needed to optimize the UABS's policy. Adopting the Continual Meta Policy Search (CoMPS) strategy, we demonstrate significant efficiency gains as compared to conventional RL, as well as to naive transfer learning methods.
Introduction
The deployment of Unmanned Aerial Vehicles (UAVs) as flying base stations (UABSs) offers a flexible solution for providing on-demand connectivity in vehicular wireless networks. These platforms are particularly valuable for vehicle-to-everything (V2X) services in environments where ground-based infrastructure is insufficient. However, a significant challenge lies in optimizing the UABS trajectory to maintain proximity to mobile vehicles and maximize network coverage.
Traditional Reinforcement Learning (RL) approaches typically train an agent from scratch for every new traffic configuration, which is computationally expensive and slow to adapt to dynamic real-world changes. While Meta-RL addresses this by "learning how to learn," it often requires maintaining a large library of simulators for all past tasks to avoid forgetting. This paper explores a more efficient alternative: continual meta-RL, which allows a UAV to improve its learning ability over time as it encounters new traffic conditions without needing to store or re-run past simulations.
Methodology: Continual Meta Policy Search (CoMPS)
The proposed approach utilizes the Continual Meta Policy Search (CoMPS) algorithm to enable online, sequential learning. CoMPS operates in two primary phases: a local RL step and a meta-update. When a new traffic configuration is encountered, the UABS initializes its policy using meta-learned parameters and performs a few episodes of standard policy gradient RL (REINFORCE) to adapt to the specific task.
Following task completion, the meta-initialization parameters are updated. To avoid the "catastrophic forgetting" common in continual learning and the high cost of re-simulating past tasks, CoMPS employs two types of memory: a Full Experience Set and a Skilled Experience Set (containing only high-reward episodes). It leverages behavioral cloning and importance sampling to refine the foundation policy, ensuring it remains an ideal starting point for future, unseen traffic patterns.
Experimental Evaluation
The performance of CoMPS was evaluated in two distinct environments:
- Toy Example: A simple grid world where vehicles move in alternating loops, testing the agent's ability to handle basic pattern changes.
- Urban Scenario: A realistic simulation of a 1500m x 900m area in Bologna, Italy, using the SUMO (Simulation of Urban MObility) traffic simulator. This scenario involved 50 sequential tasks with varying vehicle counts (15–30) and randomized movement profiles.
Results and Discussion
Experimental results demonstrate that CoMPS significantly outperforms conventional RL and simple transfer learning. In the urban scenario, the meta-learned policy achieved a much higher average number of packets collected, indicating superior coverage optimization.
Key findings include:
- Efficiency and Convergence: CoMPS showed a clear upward performance trend across tasks, effectively "learning how to learn" and reducing the time required for adaptation.
- Stability: The meta-RL approach exhibited lower variance and more consistent performance compared to baselines, which often suffered from "negative transfer" when transitioning between dissimilar tasks.
- Strategic Initialization: Over time, the meta-learned initial trajectory evolved from random movement into a strategic central path, allowing the UAV to quickly pivot toward vehicle clusters in new scenarios.
Conclusion
This work demonstrates that continual meta-RL, specifically via the CoMPS algorithm, provides a robust framework for UABS trajectory optimization in dynamic vehicular networks. By transferring knowledge from past experiences without the need for exhaustive re-simulation, the system achieves significant gains in both efficiency and coverage. Future research will focus on extending this framework to multi-UAV coordination and distributed learning architectures to further enhance scalability and real-world applicability.
Summary updated