Machine Learning Empowered Trajectory and Passive Beamforming Design in UAV-RIS Wireless Networks

Xiao Liu, Yuanwei Liu, Yue Chen

TL;DR

This paper introduces a novel framework that integrates Reconfigurable Intelligent Surfaces (RIS) into UAV-enabled wireless networks to drastically reduce drone energy consumption. By employing a Decaying Deep Q-Network (D-DQN), the system intelligently coordinates UAV movement and RIS "passive beamforming" to maintain high-quality links for mobile users, achieving 11.7% better energy efficiency than traditional orthogonal access methods.

What Problem Does This Paper Address?

Unmanned Aerial Vehicles (UAVs) are promising tools for providing "on-demand" connectivity during natural disasters or at crowded sporting events. However, they face a major hurdle: limited battery life.

In these scenarios, more than 95% of a UAV's energy is spent on propulsion—simply staying in the air and moving to maintain a Line-of-Sight (LoS) connection with mobile users. As users roam, the UAV must constantly reposition itself, draining its battery rapidly.

The authors ask: Can we use RIS—essentially "smart mirrors" for radio waves—to create virtual LoS links so the UAV doesn't have to move as much? Furthermore, how do we manage the massive complexity of optimizing the UAV's path, the RIS's reflections, and the network's power allocation all at once in a dynamic environment?

Methods

The researchers propose a joint optimization framework that handles four variables simultaneously: UAV trajectory, RIS phase shifts, power allocation, and the decoding order for Non-Orthogonal Multiple Access (NOMA).

To solve this, they developed a Decaying Deep Q-Network (D-DQN) algorithm:

  • Reinforcement Learning (RL): The central controller acts as an "agent" that learns the best actions by interacting with the dynamic environment.
  • Decaying Learning Rate: Unlike standard DQNs, this version uses a learning rate that starts high (to explore and learn quickly) and gradually decreases (to fine-tune and converge to a stable solution). This prevents the "oscillations" often seen in complex RL tasks.
  • NOMA Integration: They use NOMA to allow multiple users to share the same frequency/time resources, further boosting spectral efficiency.
  • 3D Radio Mapping: The environment is modeled using a 3D radio map of a dense urban area, accounting for signal blockages caused by buildings.

Datasets

The study does not use a static, pre-existing dataset. Instead, it utilizes a simulated environment based on:

  • 3GPP Specifications: Standardized channel models for UAV-to-ground communications.
  • 3D Urban Radio Maps: A simulated dense urban environment where buildings are modeled as sets of cubes to calculate signal blockage (NLoS).
  • User Mobility Models: Mobile users are simulated using "walk models" where their locations are updated periodically, mimicking real-world roaming patterns often derived from social media (Twitter API) data.

Key Insights and Ideas

  • RIS as a "Movement Substitute": The core "aha" moment is that adjusting the phase shifts on an RIS costs negligible energy compared to moving a multi-kilogram drone. By "steering" the signal via the RIS, the UAV can stay in a hovering state longer, preserving battery.
  • The NOMA Advantage: In dynamic environments, NOMA decoding order isn't just about who has the strongest signal; it must be re-determined at every timeslot because the RIS and the users are constantly changing the effective channel gain.
  • Stability through Decay: The authors found that standard Q-learning fails to converge because the state space (UAV position + user positions + RIS states) is too massive. The decaying learning rate in D-DQN was the key to making the system stable enough for practical use.

Key Results

  • Energy Savings: Integrating RIS into the network reduced the UAV's energy dissipation by approximately 23.3% compared to a UAV-only setup.
  • NOMA vs. OMA: The RIS-NOMA configuration consumed 11.7% less energy than the RIS-OMA (Orthogonal Multiple Access) case, proving that NOMA's spectral efficiency translates directly into hardware endurance.
  • Convergence: The D-DQN algorithm successfully converged after roughly 50,000 training episodes, whereas traditional Q-learning failed to reach a stable solution.
  • Altitude Impact: Simulations showed that higher UAV altitudes (e.g., 200m vs 100m) lead to higher energy consumption due to increased path loss and the propulsion power required to maintain higher altitudes.

Limitations and Future Work

  • CSI Acquisition: The paper assumes the system has perfect Channel State Information (CSI). In reality, RIS are passive devices that cannot easily "sense" the channel, making CSI acquisition a significant practical challenge.
  • 2D vs. 3D Trajectory: While the radio map is 3D, the UAV trajectory optimization was limited to 2D movement due to the lack of a robust energy model for complex 3D maneuvers.
  • Precoding Optimization: The authors used a standard Zero-Forcing (ZF) method for active beamforming. They suggest that future work could involve using machine learning to jointly optimize both active (UAV) and passive (RIS) beamforming for even greater gains.
  • Velocity Control: The current model focuses on position; future iterations could optimize the UAV's velocity and acceleration to further shave off propulsion energy.

Summary updated

Abstract

A novel framework is proposed for integrating reconfigurable intelligent surfaces (RIS) in unmanned aerial vehicle (UAV) enabled wireless networks, where an RIS is deployed for enhancing the service quality of the UAV. Non-orthogonal multiple access (NOMA) technique is invoked to further improve the spectrum efficiency of the network, while mobile users (MUs) are considered as roaming continuously. The energy consumption minimizing problem is formulated by jointly designing the movement of the UAV, phase shifts of the RIS, power allocation policy from the UAV to MUs, as well as determining the dynamic decoding order. A decaying deep Q-network (D-DQN) based algorithm is proposed for tackling this pertinent problem. In the proposed D-DQN based algorithm, the central controller is selected as an agent for periodically observing the state of UAV-enabled wireless network and for carrying out actions to adapt to the dynamic environment. In contrast to the conventional DQN algorithm, the decaying learning rate is leveraged in the proposed D-DQN based algorithm for attaining a tradeoff between accelerating training speed and converging to the local optimal. Numerical results demonstrate that: 1) In contrast to the conventional Q-learning algorithm, which cannot converge when being adopted for solving the formulated problem, the proposed D-DQN based algorithm is capable of converging with minor constraints; 2) The energy dissipation of the UAV can be significantly reduced by integrating RISs in UAV-enabled wireless networks; 3) By designing the dynamic decoding order and power allocation policy, the RIS-NOMA case consumes 11.7% less energy than the RIS-OMA case.
BibTeX
@article{liu2021machine,
  author = {Xiao Liu and Yuanwei Liu and Yue Chen},
  title = {{Machine Learning Empowered Trajectory and Passive Beamforming Design in UAV-RIS Wireless Networks}},
  journal = {IEEE J. Sel. Areas Commun.},
  year = {2021},
  doi = {10.1109/jsac.2020.3041401},
}

Loading related content...

Figure