Continual Meta-Reinforcement Learning for UAV-Aided Vehicular Wireless Networks

Riccardo Marini, Sangwoo Park, Osvaldo Simeone, Chiara Buratti

TL;DR

This paper introduces a continual meta-reinforcement learning approach to help Unmanned Aerial Vehicles (UAVs) optimize their flight paths for maximum network coverage in dynamic vehicular environments. By using the Continual Meta Policy Search (CoMPS) algorithm, the system learns to transfer knowledge from past traffic patterns to new ones, significantly speeding up adaptation without needing to revisit old simulations.

What Problem Does This Paper Address?

Deploying UAVs as "flying base stations" (UABSs) is a promising way to provide on-demand connectivity for vehicle-to-everything (V2X) services, especially in areas with poor ground coverage. However, finding the optimal flight path to stay near moving vehicles is a moving target.

The standard approach uses Reinforcement Learning (RL), where a UAV learns through trial and error. The problem? Most RL agents start from scratch every time the traffic pattern changes (e.g., a new rush hour or a different road layout). While "Meta-RL" exists to help agents learn how to learn, it usually requires a massive library of simulators for every past scenario to be constantly available. This paper tackles the challenge of online, sequential learning: how can a UAV improve its "learning ability" over time as it encounters new traffic conditions, without needing to store or re-run simulations of the past?

Methods

The authors leverage Continual Meta Policy Search (CoMPS). Here’s how the "Meta" and "Continual" parts work together:

  1. The Local RL Step: When faced with a new traffic configuration, the UAV starts with a "meta-learned" initial policy (the starting weights of its neural network). It then runs a standard policy gradient RL (REINFORCE) for a few episodes to fine-tune its trajectory for that specific task.
  2. The Meta-Update (The "Secret Sauce"): After completing a task, the system updates its initialization parameters. It uses two types of memory:
  3. Full Experience Set: A record of all actions and rewards from the current task.
  4. Skilled Experience Set: A record of only the "best" (highest reward) episodes.
  5. Off-Policy Learning & Behavioral Cloning: To update the meta-initialization without re-running old simulators, CoMPS uses behavioral cloning (trying to mimic the "skilled" trajectories of the past) and importance sampling (adjusting for the fact that the old data was collected using a different version of the policy).

This allows the UAV to build a "foundation" policy that is specifically designed to be easy to adapt to any new traffic pattern it might encounter.

Datasets

The study evaluates the approach using two main setups:

  • Toy Example: A simple 40m x 40m grid world where three vehicles move in either clockwise or counter-clockwise loops. This tests the agent's ability to handle basic alternating patterns.
  • Urban Scenario: A realistic simulation of a 1500m x 900m area in Bologna, Italy. The authors used SUMO (Simulation of Urban MObility), a professional-grade traffic simulator, to generate 50 different task configurations with varying numbers of vehicles (15–30) and random speeds/paths.

Key Insights and Ideas

  • Avoiding "Negative Transfer": Simple transfer learning (just starting the next task with the final weights of the previous one) often fails. If Task A and Task B are very different, the knowledge from A can actually hinder the UAV in Task B. Meta-RL avoids this by learning a flexible "middle ground" starting point rather than a specialized end point.
  • Efficiency via Memory, Not Simulation: The biggest hurdle in Meta-RL is the computational cost of revisiting old tasks. By storing "skilled experiences" and using offline RL techniques, CoMPS achieves meta-learning benefits with a fraction of the computational overhead.
  • Trajectory "Identification": As the meta-learning progresses, the initial (pre-adaptation) trajectory of the UAV evolves from a random path into a central, strategic route that allows it to quickly "pivot" toward wherever the vehicles are in a new task.

Key Results

  • Superior Packet Collection: In the Bologna urban scenario, the Meta-RL approach (CoMPS) consistently collected significantly more V2X packets than both conventional RL and basic transfer learning.
  • Faster Convergence: While conventional RL struggled to improve across 50 different tasks, CoMPS showed a clear upward trend in performance. It effectively "learned how to learn," becoming more efficient at maximizing coverage as it encountered more diverse traffic patterns.
  • Stability: The meta-learned policy showed lower variance (more consistent performance) compared to the baselines, which often suffered from erratic performance when faced with unfamiliar vehicle movements.

Limitations and Future Work

  • Single UAV Focus: The current model assumes a single UAV. In real-world deployments, multiple UAVs would need to coordinate their trajectories to avoid overlaps and gaps in coverage.
  • Centralized Learning: The paper assumes a centralized controller manages the learning. Moving toward distributed meta-learning, where UAVs share knowledge directly with each other, is a suggested next step.
  • Sim-to-Real Gap: While SUMO provides realistic traffic, the paper does not address the "sim-to-real" transition, where physical factors like wind, battery constraints, and hardware latency might affect the RL policy.

Summary updated

Abstract

Unmanned aerial base stations (UABSs) can be deployed in vehicular wireless networks to support applications such as extended sensing via vehicle-to-everything (V2X) services. A key problem in such systems is designing algorithms that can efficiently optimize the trajectory of the UABS in order to maximize coverage. In existing solutions, such optimization is carried out from scratch for any new traffic configuration, often by means of conventional reinforcement learning (RL). In this paper, we propose the use of continual meta-RL as a means to transfer information from previously experienced traffic configurations to new conditions, with the goal of reducing the time needed to optimize the UABS's policy. Adopting the Continual Meta Policy Search (CoMPS) strategy, we demonstrate significant efficiency gains as compared to conventional RL, as well as to naive transfer learning methods.
BibTeX
@inproceedings{marini2023continual,
  author = {Riccardo Marini and Sangwoo Park and Osvaldo Simeone and Chiara Buratti},
  title = {{Continual Meta-Reinforcement Learning for UAV-Aided Vehicular Wireless Networks}},
  booktitle = {ICC},
  year = {2023},
  url = {https://openreview.net/forum?id=GnfdM7K8cY},
}

Loading related content...

Figure