UAV-Aided Cellular Communications with Deep Reinforcement Learning Against Jamming

Xiaozhen Lu, Liang Xiao, Canhuang Dai, Huaiyu DaiIEEE Wirel. Commun 2020 · DOI

TL;DR

This paper introduces a novel UAV-aided cellular communication system that uses deep reinforcement learning (DRL) to dynamically optimize relay power against sophisticated "smart" jamming attacks. By leveraging transfer learning and a deep convolutional neural network (CNN), the system learns optimal anti-jamming strategies without prior knowledge of the jammer's model or network conditions, significantly reducing bit error rates and saving energy.

What Problem Does This Paper Address?

Modern cellular systems, especially those supporting demanding applications like augmented reality, are increasingly vulnerable to jamming attacks. These attacks are becoming more sophisticated, with "smart" jammers capable of adapting their jamming strategies (e.g., channel selection, power) based on observed communication policies and network states. This poses a significant threat to network reliability, user privacy, and battery life. While Unmanned Aerial Vehicles (UAVs) offer a promising solution due to their mobility and line-of-sight capabilities, existing UAV-aided relay schemes often require detailed knowledge of network and jamming models, which is impractical in dynamic, adversarial environments. This paper aims to bridge this gap by developing an intelligent UAV relay system that can autonomously learn and adapt to smart jamming attacks.

Methods

The core of this paper is a Deep Reinforcement Learning (DRL) based UAV Relay Scheme (DRLUR). The system employs a UAV that acts as a relay for mobile users when their serving base station (BS) is jammed. The UAV's task is to decide its optimal relay power.

Key techniques include:

  • Deep Reinforcement Learning (DRL): The problem is framed as a Markov Decision Process (MDP), where the UAV learns an optimal policy through trial and error.
  • Deep Convolutional Neural Network (CNN): A CNN is used to process the high-dimensional state space (including BER, channel gains, and estimated jamming power) and compress it into a manageable representation for the DRL agent.
  • Transfer Learning (Hotbooting): To accelerate the learning process and overcome the challenges of initial random exploration, the CNN weights are initialized using experiences from similar previous anti-jamming relay scenarios. This "hotbooting" technique significantly reduces learning time.
  • Experience Replay: The system stores past experiences (state, action, reward, next state) in a memory pool and samples from it to train the CNN, improving learning stability and efficiency.
  • E-greedy Algorithm: This exploration-exploitation strategy is used to select the relay power, balancing the need to explore new strategies with exploiting known good ones to avoid local optima.

The system learns to optimize relay power without explicit knowledge of the jammer's model or the exact network topology, relying instead on observed BER, channel conditions, and estimated jamming power.

Datasets

Not discussed in the paper. The paper focuses on simulation-based evaluation rather than training on pre-existing datasets.

Key Insights and Ideas

  • Autonomous Adaptation to Smart Jammers: The most significant insight is the ability of the DRLUR scheme to effectively counter sophisticated, adaptive jamming strategies without needing to know the jammer's underlying model. This is crucial for real-world deployment where jammer behavior is often unknown.
  • Accelerated Learning via Transfer Learning: The use of transfer learning to initialize CNN weights is a critical innovation. It dramatically reduces the time and number of interactions required for the UAV to learn an effective anti-jamming policy, making the system practical for dynamic environments.
  • State Compression for Complex Environments: Employing CNNs to compress the state space is key to handling the complexity of real-world cellular environments, where numerous factors influence the optimal relay decision.
  • Performance Beyond Benchmarks: The DRLUR scheme not only matches but significantly outperforms existing methods like HPUR and Q-learning in terms of BER reduction and energy savings, especially under challenging conditions.

Key Results

  • Reduced Bit Error Rate (BER): The DRLUR scheme significantly reduces the BER of user messages. For example, it achieved a reduction of 46.6% compared to HPUR and 99.7% compared to Q-learning at the 1000th time slot in one simulation case.
  • Energy Savings: The system demonstrates substantial energy efficiency improvements. It saved 33.6% of cellular energy compared to HPUR.
  • Faster Learning: The DRLUR scheme converges much faster than benchmark methods. It took approximately 200 time slots (0.22s) to optimize relay power, saving 84.6% of the time required by HPUR.
  • Robustness: The proposed scheme is less sensitive to state estimation errors and delays compared to prior methods.

Limitations and Future Work

  • State Estimation Error and Delay: The authors acknowledge that the current analysis assumes accurate state estimation, which is not always true in practical UAV-aided cellular systems. Future work will focus on designing DRL schemes that are robust to state estimation errors and delays.
  • "Bad" Policies: Deep learning techniques like DQN require exploration, which can involve trying "bad" policies during the learning process. This can lead to network disruptions or failures to send critical information. The paper suggests that backup anti-jamming protocols should be designed and incorporated to mitigate this risk.
  • Computational Complexity: While CNNs compress the state space, their computational complexity can still be a factor. The paper provides an expression for the complexity but doesn't delve deeply into optimization for extremely resource-constrained UAVs.

Summary updated

BibTeX
@article{lu2020uav,
  author = {Xiaozhen Lu and Liang Xiao and Canhuang Dai and Huaiyu Dai},
  title = {{UAV-Aided Cellular Communications with Deep Reinforcement Learning Against Jamming}},
  journal = {IEEE Wirel. Commun.},
  year = {2020},
  doi = {10.1109/mwc.001.1900207},
}
Figure