Real-Time Energy Harvesting Aided Scheduling in UAV-Assisted D2D Networks Relying on Deep Reinforcement Learning

Khoi Khac Nguyen, Ngo Anh Vien, Long D. Nguyen, Minh-Tuan Le, Lajos Hanzo, Trung Q. Duong

Abstract

Unmanned aerial vehicle (UAV)-assisted device-to-device (D2D) communications can be deployed flexibly thanks to UAVs' agility. By exploiting the direct D2D interaction supported by UAVs, both the user experience and network performance can be substantially enhanced at public events. However, the continuous moving of D2D users, limited energy and flying time of UAVs are impediments to their applications in real-time. To tackle this issue, we propose a novel model based on deep reinforcement learning in order to find the optimal solution for the energy-harvesting time scheduling in UAV-assisted D2D communications. To make the system model more realistic, we assume that the UAV flies around a central point, the D2D users move continuously with random walk model and the channel state information encountered during each time slot is randomly time-variant. Our numerical results demonstrate that the proposed schemes outperform the existing solutions. The associated energy efficiency game can be solved in less than one millisecond by an off-the-shelf processor using trained neural networks. Hence our deep reinforcement learning techniques are capable of solving real-time resource allocation problems in UAV-assisted wireless networks.

Introduction

Next-generation wireless networks increasingly rely on Unmanned Aerial Vehicles (UAVs) to provide temporary coverage in high-demand or disaster-stricken areas. These UAVs facilitate Device-to-Device (D2D) communication, allowing users to transmit data directly to one another. However, managing such networks in real-time presents significant challenges due to the limited battery life of UAVs and the dynamic nature of mobile users.

Traditional resource allocation methods, such as Lagrangian relaxation, often fail in these scenarios because their high computational latency cannot keep pace with rapidly changing channel states and user mobility. This paper introduces a Deep Reinforcement Learning (DRL) framework designed to optimize energy harvesting and data transmission schedules instantaneously.

System Model and Methodology

The researchers modeled the network as a Markov Decision Process (MDP) where a UAV serves multiple D2D pairs. The system operates on a "Harvest-then-Transmit" protocol: in each time slot, D2D transmitters first harvest energy from the UAV's downlink signal before using that energy for peer-to-peer data transmission.

System model of the UAV-assisted D2D network featuring energy harvesting (EH) links and inter-device interference.

To solve the scheduling problem, two DRL architectures were evaluated:

  • Deep Deterministic Policy Gradient (DDPG): An actor-critic algorithm suited for continuous action spaces, allowing for precise tuning of the harvesting time ($\tau$).
  • Proximal Policy Optimization (PPO): Implemented with Clipping and KL Divergence Penalty variants to ensure stable policy updates in volatile environments.

The simulation environment, built using TensorFlow 1.13.1 and CVXPY, accounted for realistic factors including a "Random Walk" mobility model for 30 D2D pairs, a 1 MHz bandwidth, and a 5W UAV transmit power. The UAV was modeled as moving around a central point, simulating the effects of wind and flight dynamics.

Key Insights and Results

The study's most critical finding is the dramatic reduction in inference time. While traditional Optimal Harvesting Time (OHT) optimization requires between 54ms and 170ms to find a solution, the DRL-based models achieve this in just 0.2ms to 0.3ms. This 200x to 500x speedup enables true real-time resource allocation.

Furthermore, the DRL approaches demonstrated:

  • Superior Energy Efficiency (EE): Consistently outperforming the OHT benchmark across various Quality of Service (QoS) requirements and D2D pair densities.
  • Robust Convergence: The models reached stable, high-performing policies within approximately 50 training episodes.
  • Algorithm Performance: PPO generally converged faster and proved more robust than DDPG, likely due to its superior handling of the fluctuating channel quality inherent in mobile wireless environments.

Conclusion and Future Directions

By shifting the computational burden to an offline training phase, DRL allows UAV-assisted networks to adapt to environmental changes in sub-millisecond timeframes. While this study focused on a single UAV with random-walk mobility, it lays the groundwork for more complex scenarios. Future research will explore joint optimization frameworks that integrate 3D trajectory planning, power allocation, and multi-UAV coordination to further enhance the resilience and efficiency of aerial wireless infrastructure.

Summary updated

Abstract

Unmanned aerial vehicle (UAV)-assisted device-to-device (D2D) communications can be deployed flexibly thanks to UAVs' agility. By exploiting the direct D2D interaction supported by UAVs, both the user experience and network performance can be substantially enhanced at public events. However, the continuous moving of D2D users, limited energy and flying time of UAVs are impediments to their applications in real-time. To tackle this issue, we propose a novel model based on deep reinforcement learning in order to find the optimal solution for the energy-harvesting time scheduling in UAV-assisted D2D communications. To make the system model more realistic, we assume that the UAV flies around a central point, the D2D users move continuously with random walk model and the channel state information encountered during each time slot is randomly time-variant. Our numerical results demonstrate that the proposed schemes outperform the existing solutions. The associated energy efficiency game can be solved in less than one millisecond by an off-the-shelf processor using trained neural networks. Hence our deep reinforcement learning techniques are capable of solving real-time resource allocation problems in UAV-assisted wireless networks.
BibTeX
@inproceedings{nguyen2020real,
  author = {Khoi Khac Nguyen and Ngo Anh Vien and Long D. Nguyen and Minh-Tuan Le and Lajos Hanzo and Trung Q. Duong},
  title = {{Real-Time Energy Harvesting Aided Scheduling in UAV-Assisted D2D Networks Relying on Deep Reinforcement Learning}},
  booktitle = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3046499},
}
Figure