Integrating LEO Satellite and UAV Relaying via Reinforcement Learning for Non-Terrestrial Networks

Ju-Hyung Lee, Jihong Park, Mehdi Bennis, Young-Chai KoarXiv 2020 · arXiv

TL;DR

This paper introduces a Deep Reinforcement Learning (DRL) framework to optimize long-range non-terrestrial networks by jointly managing LEO satellite selection and mobile UAV relay trajectories. By incorporating a novel action-dimension reduction technique, the proposed Deep Q-Network (DQN) successfully navigates the massive, time-varying topology of satellite mega-constellations, delivering up to a 5.74x increase in average data rate compared to direct ground communication.

What Problem Does This Paper Address?

Next-generation wireless networks (Beyond 5G/6G) aim to provide seamless, three-dimensional connectivity across the globe using Non-Terrestrial Networks (NTNs). A promising architecture involves forwarding data between distant ground terminals by relaying signals through Low Earth Orbit (LEO) satellite mega-constellations and mobile High-Altitude Platforms (HAPs), such as fixed-wing unmanned aerial vehicles (UAVs).

However, optimizing this multi-hop network is incredibly challenging. LEO satellites orbit the Earth at high speeds, creating a rapidly changing network topology. Furthermore, commercial mega-constellations (like SpaceX's Starlink or OneWeb) feature thousands of satellites, resulting in an enormous, high-dimensional action space. Traditional optimization methods like Successive Convex Approximation (SCA) or standard dynamic programming become computationally intractable under these highly dynamic, large-scale conditions.

Methods

To overcome the modeling complexities and massive scale, the authors employ a model-free Deep Reinforcement Learning (DRL) approach based on a Deep Q-Network (DQN).

The problem is formulated as a Markov Decision Process (MDP):

  • State Space: Includes the time-varying positions of the LEO satellites and the HAP, the relative link distances, and the instantaneous achievable data rates across each hop.
  • Action Space: A joint action consisting of the satellite association decision (which satellite to connect to, using one-hot encoding) and the HAP's acceleration vector (discretized to form a finite action space).
  • Reward Function: A normalized sigmoid function designed to reward actions that outperform a fixed-position baseline and penalize those that underperform.
Figure 2: DQN structure for vertical multi-hop communication in LEO satellite constellation network.

Novelty: Action Dimension Reduction

The core innovation that makes the DQN viable in a mega-constellation environment is an action dimension reduction technique. Instead of evaluating every satellite in the entire orbital plane, the algorithm restricts its view to a localized coordinate space between the source and destination terminals. By applying a modulo operation based on the localized space size, the algorithm prunes distant satellites from the action space, dramatically accelerating training convergence.

Datasets

The paper does not utilize a static, real-world dataset. Instead, the authors evaluate their approach using a simulated environment modeled on realistic aerospace and wireless communication parameters:

  • LEO Constellation: 22 satellites distributed evenly across a single orbital plane at an altitude of 550 km, traveling at a stable orbital velocity of 7.8 km/s.
  • Simulation Area: A 3D coordinate space spanning $4000 \times 4000 \times 550 \text{ km}^3$.
  • Channel Model: Deterministic line-of-sight (LoS) propagation models for the satellite-to-aerial and aerial-to-ground RF links.

Key Insights and Ideas

  • Bottleneck (Weakest Link) Optimization: Because the system uses a bufferless decode-and-forward protocol, the end-to-end data rate is limited by the worst-performing hop. The DQN successfully learns to optimize the system holistically rather than trying to maximize a single high-performing link.
  • Counter-Intuitive Routing: The optimal policy sometimes chooses not to connect to the closest satellite. Due to information-causality constraints, connecting to a slightly further satellite can occasionally yield a better balanced multi-hop data rate.
  • Emergent Oval Trajectories: When optimizing the HAP's path, the DRL agent naturally learns an oval-like flight trajectory. This movement pattern allows the HAP to dynamically balance its distance between the rapidly moving LEO satellite and the stationary ground destination.
Optimized Src-SAT-HAP associations and HAP trajectory during time slots.

Key Results

The simulation results demonstrate clear performance gains over standard benchmarks:

  • Spectral Efficiency: The proposed mobile HAP-relaying scheme achieves a spectral efficiency of $2.5862 \times 10^{-3} \text{ bits/Hz}$.
  • Data Rate Gains: This represents a 5.74x (~298.61%) throughput gain over direct ground-to-ground transmission, and a 3.99x gain over using an LEO satellite network without an additional HAP relay.
  • Mobility Advantage: Active HAP trajectory control yields a 13.04% gain over a baseline that uses a fixed-position HAP relay, validating the importance of real-time mobility management.
Figure 5: End-to-end data rate for different relaying schemes.

Limitations and Future Work

  • Single Orbit Simplification: The current model evaluates only a single orbital plane. Real-world mega-constellations feature multiple intersecting orbital planes, which will significantly complicate the satellite handover process.
  • Edge Effects: The authors noted "edge effects" (performance fluctuations) toward the end of simulation episodes due to the finite time-horizon setup.
  • Scalability to Multi-Link Scenarios: A single-agent DQN introduces high overhead if scaled to multiple ground terminals. The authors point to Multi-Agent Reinforcement Learning (MARL) and distributed Actor-Critic architectures as essential next steps to make this approach practical for dense, real-world networks.

Summary updated

Abstract

A mega-constellation of low-earth orbit (LEO) satellites has the potential to enable long-range communication with low latency. Integrating this with burgeoning unmanned aerial vehicle (UAV) assisted non-terrestrial networks will be a disruptive solution for beyond 5G systems provisioning large scale three-dimensional connectivity. In this article, we study the problem of forwarding packets between two faraway ground terminals, through an LEO satellite selected from an orbiting constellation and a mobile high-altitude platform (HAP) such as a fixed-wing UAV. To maximize the end-to-end data rate, the satellite association and HAP location should be optimized, which is challenging due to a huge number of orbiting satellites and the resulting time-varying network topology. We tackle this problem using deep reinforcement learning (DRL) with a novel action dimension reduction technique. Simulation results corroborate that our proposed method achieves up to 5.74x higher average data rate compared to a direct communication baseline without SAT and HAP.
BibTeX
@misc{lee2020integrating,
  author = {Ju-Hyung Lee and Jihong Park and Mehdi Bennis and Young-Chai Ko},
  title = {{Integrating LEO Satellite and UAV Relaying via Reinforcement Learning for Non-Terrestrial Networks}},
  year = {2020},
  eprint = {2005.12521},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.12521},
}

Loading related content...

Figure