Novel Online-Offline MA2C-DDPG for Efficient Spectrum Allocation and Trajectory Optimization in Dynamic Spectrum Sharing UAV Networks

Rui Ding, Fuhui Zhou, Yuben Qu, Chao Dong, Qihui Wu, Tony Q. S. QuekICCC 2023

TL;DR

This paper tackles the challenge of efficiently allocating spectrum and optimizing flight paths for Unmanned Aerial Vehicle (UAV) networks operating in crowded, unlicensed bands, especially when facing jamming. They propose a novel "online-offline MA2C-DDPG" framework that combines the strengths of online learning for spectrum decisions and offline learning for trajectory planning, achieving superior performance and faster convergence compared to existing methods.

What Problem Does This Paper Address?

Unmanned Aerial Vehicles (UAVs) are increasingly vital for applications like communication relays and emergency services due to their flexibility. However, they face two major hurdles: spectrum scarcity in unlicensed bands and vulnerability to jamming. This means UAVs must efficiently share limited spectrum with other users and avoid interference, all while navigating dynamically. Traditional optimization methods struggle with the complexity and non-convexity of these joint problems, often yielding suboptimal solutions. Machine learning, particularly Deep Reinforcement Learning (DRL), offers a promising path, but existing DRL approaches have drawbacks: online learning can be data-inefficient and unstable in dynamic environments, while offline learning requires extensive data and can be slow to adapt. This paper aims to bridge this gap by developing a hybrid DRL approach that leverages the best of both worlds for robust spectrum allocation and trajectory optimization in the face of jamming.

Methods

The core innovation is a Novel Online-Offline MA2C-DDPG framework. This framework ingeniously combines two DRL techniques:

  • Online Multi-Agent Actor-Critic (MA2C): This is used for spectrum allocation. MA2C is chosen for its ability to handle large, discrete action spaces (deciding which spectrum band to use) and its adaptability to dynamic environments. It employs a decentralized multi-agent strategy where each UAV (or secondary user) acts independently but receives a global reward, improving system-wide performance and reducing action space complexity.
  • Offline Deep Deterministic Policy Gradient (DDPG): This is employed for UAV trajectory optimization. DDPG is favored for continuous action spaces (determining the UAV's flight path) and its data efficiency, making it suitable for optimizing the UAV's movement over time. It utilizes an experience replay strategy to learn from historical data.

This hybrid approach allows the system to make real-time spectrum allocation decisions while learning optimal flight paths from past experiences, creating a synergistic effect.

Datasets

Not discussed in the paper. The paper focuses on simulation-based evaluation using a defined system model.

Key Insights and Ideas

  • Hybrid Learning for Complementary Problems: The paper demonstrates that combining online learning (MA2C) for discrete, reactive tasks (spectrum allocation) and offline learning (DDPG) for continuous, long-term planning (trajectory optimization) is a powerful strategy for complex UAV network management.
  • MA2C's Advantage in Dynamic Environments: The MA2C component proves effective in adapting to the rapidly changing spectrum conditions and jamming threats, outperforming single-agent approaches that struggle with the increased complexity.
  • DDPG's Efficiency for Trajectory Planning: Offline DDPG's data efficiency is crucial for optimizing UAV flight paths, where collecting extensive real-world data can be impractical.
  • Overcoming Jamming: The proposed framework shows significant resilience and improved performance even under various jamming modes (random, sweep, constant), highlighting its robustness.

Key Results

  • Superior Performance: The proposed MA2C-DDPG framework consistently achieved the largest sum transmission rate for the secondary network compared to benchmark schemes (SA2C-DDPG, random spectrum allocation, and random UAV trajectory).
  • Faster Convergence: The framework demonstrated a faster convergence speed in training episodes, reaching stable rewards more quickly than the SA2C-DDPG benchmark.
  • Improved Reward: The MA2C-DDPG achieved a higher converged reward, indicating better overall system performance and efficiency.
  • Resilience to Jamming: Performance improvements were particularly pronounced in the random jamming UAV mode, showcasing the framework's ability to adapt to unpredictable threats.
  • Scalability: The sum transmission rate increased with the number of secondary users (SUs), indicating good scalability.

Limitations and Future Work

  • Limitations: The paper acknowledges that the single-agent based method (SA2C-DDPG) faces challenges due to the exponentially increased action space complexity, making exploration more difficult. The paper also implicitly suggests that the performance might be sensitive to the specific parameters of the DRL algorithms and the simulation environment.
  • Future Work: While not explicitly detailed, the success of this hybrid approach suggests future work could explore:
  • Applying this framework to more complex multi-UAV scenarios with diverse communication requirements.
  • Investigating more sophisticated jamming models and counter-jamming strategies.
  • Exploring transfer learning or meta-learning to further accelerate adaptation to new environments or jamming patterns.
  • Real-world deployment and validation of the proposed framework.

Summary updated

Abstract

Unmanned aerial vehicle (UAV) communication is of crucial importance for diverse practical applications. However, it is susceptible to the severe spectrum scarcity problem and interference since it operates in the unlicensed spectrum band. In order to tackle those issues, a dynamic spectrum sharing network is considered with the anti-jamming technique. Moreover, an intelligent spectrum allocation and trajectory optimization scheme is proposed to adapt to diverse jamming models by exploiting our designed novel online-offline multi-agent actor-critic and deep deterministic policy-gradient framework. Simulation results demonstrate the high efficiency of our proposed framework. It is also shown that our proposed scheme achieves the largest transmission rate among all benchmark schemes.
BibTeX
@inproceedings{ding2023novel,
  author = {Rui Ding and Fuhui Zhou and Yuben Qu and Chao Dong and Qihui Wu and Tony Q. S. Quek},
  title = {{Novel Online-Offline MA2C-DDPG for Efficient Spectrum Allocation and Trajectory Optimization in Dynamic Spectrum Sharing UAV Networks}},
  booktitle = {ICCC},
  year = {2023},
}
Figure