Multi-agent deep reinforcement learning for RIS-assisted secure UAV communication

B. Zhao, Danyang Qin, Yuhong Chen, Jiaqiang Yang, Huapeng Tang, Lin MaJournal of King Saud University - Computer and Information Sciences 2025 · DOI

Introduction

As the telecommunications industry moves toward 6G, Unmanned Aerial Vehicles (UAVs) are increasingly deployed as aerial access points to provide network services in dense urban environments or disaster zones. While UAVs offer high mobility and flexibility, their air-to-ground communication links are predominantly Line-of-Sight (LoS), making them highly susceptible to interception by ground-based eavesdroppers. Traditional cryptographic security measures can be computationally prohibitive for resource-constrained IoT devices. Consequently, Physical Layer Security (PLS) has emerged as a promising alternative, leveraging the physical properties of the wireless channel to ensure secure transmission.

System Model and Problem Formulation

To enhance PLS, this paper proposes a UAV-ground secure communication system assisted by a Reconfigurable Intelligent Surface (RIS). The RIS, composed of many passive reflecting elements, is deployed on building facades to intelligently reflect signals, thereby improving the signal quality for legitimate users while degrading it for eavesdroppers.

System model of the RIS-assisted UAV communication network.

The research objective is to maximize the Sum Secrecy Rate (SSR) and Secrecy Energy Efficiency (SEE) of the system. This involves the joint optimization of several coupled variables: the UAV's flight trajectory, active beamforming at the UAV, passive beamforming at the RIS, and transmission power. Due to the dynamic nature of the environment and the non-convexity of the optimization problem, traditional methods struggle to provide real-time solutions.

The MATD3-EP Algorithm

The authors introduce the Multi-Agent Twin Delayed Deep Deterministic Policy Gradient with Energy Penalty (MATD3-EP) algorithm. This framework addresses the optimization challenge by decomposing it into manageable subtasks handled by multiple agents.

Architecture of the proposed MATD3-EP algorithm.

The algorithm incorporates three key innovations:

  1. Multi-Agent Cooperation: Instead of a single complex controller, the system uses multiple agents that share a global reward. This decoupling prevents "inter-task interference," where optimizing one parameter (like trajectory) might negatively impact another (like beamforming).
  2. Twin Delayed DDPG (TD3): TD3 improves upon standard DDPG by using two critic networks to estimate the Q-value, taking the minimum of the two to prevent overestimation bias. This results in significantly more stable training and better convergence.
  3. Energy Penalty (EP) Mechanism: To address the limited battery life of UAVs, the reward function includes a dynamic energy penalty. This penalty scales based on the current secrecy performance, forcing the agents to find a balance between maximizing security and minimizing propulsion and transmission energy.

Performance Evaluation

The proposed MATD3-EP algorithm was tested in a simulated 3D environment using the 3D Spatial Variation (SV) channel model at a 28 GHz carrier frequency.

The simulation results demonstrate that MATD3-EP significantly outperforms benchmark strategies:

  • Secrecy Performance: The algorithm achieved a steady SSR of approximately 5 bit/s/Hz, which is more stable and higher than standard MATD3 or PPO-based methods.
  • Energy Efficiency: By incorporating the energy penalty, the system reduced total energy consumption to 11.01 KJ, a substantial improvement over the 14.70 KJ consumed by the D-PPO benchmark.
  • Trajectory Intelligence: The UAV learned to fly in a pattern that maximizes the distance from eavesdroppers while staying within the "reflection zone" of the RIS to maintain high-quality links with legitimate users.

Conclusion and Future Directions

This work highlights the potential of combining RIS technology with multi-agent reinforcement learning to solve complex security and energy challenges in UAV-assisted networks. While the current model assumes a fixed flight altitude and perfect RIS phase shifts, future research will focus on 3D trajectory optimization, hardware-constrained RIS modeling, and scenarios with imperfect Channel State Information (CSI) to further bridge the gap between simulation and real-world deployment.

Summary updated

Abstract

Unmanned aerial vehicles (UAVs) serve as aerial access points or base stations, providing network services for IoT devices in urban environments with dense infrastructure. However, the line-of-sight nature of air-ground communication links renders signal transmission vulnerable to interference from ground-based eavesdroppers. To address this challenge, we propose a UAV-ground secure communication system assisted by a reconfigurable intelligent surface (RIS). The RIS is deployed on building facades to enhance the transmission environment and improve the received signal quality for legitimate users. Our goal is to maximize the sum secrecy rate (SSR) of the communication system by jointly optimizing the UAV flight trajectory, active and passive beamforming, and transmission power. Additionally, the impact of system energy consumption is considered in the optimization process to improve secrecy energy efficiency (SEE). To achieve this, we develop a multi-agent twin delayed deep deterministic policy gradient algorithm with energy penalty (MATD3-EP), where multiple agents dynamically solve subproblems and share rewards to achieve global optimization. Simulation results demonstrate that the proposed algorithm significantly outperforms benchmark strategies in maximizing both average SSR and SEE.
BibTeX
@article{zhao2025multi,
  author = {B. Zhao and Danyang Qin and Yuhong Chen and Jiaqiang Yang and Huapeng Tang and Lin Ma},
  title = {{Multi-agent deep reinforcement learning for RIS-assisted secure UAV communication}},
  journal = {Journal of King Saud University - Computer and Information Sciences},
  year = {2025},
  doi = {10.1007/s44443-025-00145-9},
}
Figure