TL;DR
This paper introduces a Deep Reinforcement Learning (DRL) framework to optimize long-range non-terrestrial networks by jointly managing LEO satellite selection and mobile UAV relay trajectories. By incorporating a novel action-dimension reduction technique, the proposed Deep Q-Network (DQN) successfully navigates the massive, time-varying topology of satellite mega-constellations, delivering up to a 5.74x increase in average data rate compared to direct ground communication.
What Problem Does This Paper Address?
Next-generation wireless networks (Beyond 5G/6G) aim to provide seamless, three-dimensional connectivity across the globe using Non-Terrestrial Networks (NTNs). A promising architecture involves forwarding data between distant ground terminals by relaying signals through Low Earth Orbit (LEO) satellite mega-constellations and mobile High-Altitude Platforms (HAPs), such as fixed-wing unmanned aerial vehicles (UAVs).
However, optimizing this multi-hop network is incredibly challenging. LEO satellites orbit the Earth at high speeds, creating a rapidly changing network topology. Furthermore, commercial mega-constellations (like SpaceX's Starlink or OneWeb) feature thousands of satellites, resulting in an enormous, high-dimensional action space. Traditional optimization methods like Successive Convex Approximation (SCA) or standard dynamic programming become computationally intractable under these highly dynamic, large-scale conditions.
Methods
To overcome the modeling complexities and massive scale, the authors employ a model-free Deep Reinforcement Learning (DRL) approach based on a Deep Q-Network (DQN).
The problem is formulated as a Markov Decision Process (MDP):
- State Space: Includes the time-varying positions of the LEO satellites and the HAP, the relative link distances, and the instantaneous achievable data rates across each hop.
- Action Space: A joint action consisting of the satellite association decision (which satellite to connect to, using one-hot encoding) and the HAP's acceleration vector (discretized to form a finite action space).
- Reward Function: A normalized sigmoid function designed to reward actions that outperform a fixed-position baseline and penalize those that underperform.
Novelty: Action Dimension Reduction
The core innovation that makes the DQN viable in a mega-constellation environment is an action dimension reduction technique. Instead of evaluating every satellite in the entire orbital plane, the algorithm restricts its view to a localized coordinate space between the source and destination terminals. By applying a modulo operation based on the localized space size, the algorithm prunes distant satellites from the action space, dramatically accelerating training convergence.
Datasets
The paper does not utilize a static, real-world dataset. Instead, the authors evaluate their approach using a simulated environment modeled on realistic aerospace and wireless communication parameters:
- LEO Constellation: 22 satellites distributed evenly across a single orbital plane at an altitude of 550 km, traveling at a stable orbital velocity of 7.8 km/s.
- Simulation Area: A 3D coordinate space spanning $4000 \times 4000 \times 550 \text{ km}^3$.
- Channel Model: Deterministic line-of-sight (LoS) propagation models for the satellite-to-aerial and aerial-to-ground RF links.
Key Insights and Ideas
- Bottleneck (Weakest Link) Optimization: Because the system uses a bufferless decode-and-forward protocol, the end-to-end data rate is limited by the worst-performing hop. The DQN successfully learns to optimize the system holistically rather than trying to maximize a single high-performing link.
- Counter-Intuitive Routing: The optimal policy sometimes chooses not to connect to the closest satellite. Due to information-causality constraints, connecting to a slightly further satellite can occasionally yield a better balanced multi-hop data rate.
- Emergent Oval Trajectories: When optimizing the HAP's path, the DRL agent naturally learns an oval-like flight trajectory. This movement pattern allows the HAP to dynamically balance its distance between the rapidly moving LEO satellite and the stationary ground destination.
Key Results
The simulation results demonstrate clear performance gains over standard benchmarks:
- Spectral Efficiency: The proposed mobile HAP-relaying scheme achieves a spectral efficiency of $2.5862 \times 10^{-3} \text{ bits/Hz}$.
- Data Rate Gains: This represents a 5.74x (~298.61%) throughput gain over direct ground-to-ground transmission, and a 3.99x gain over using an LEO satellite network without an additional HAP relay.
- Mobility Advantage: Active HAP trajectory control yields a 13.04% gain over a baseline that uses a fixed-position HAP relay, validating the importance of real-time mobility management.
Limitations and Future Work
- Single Orbit Simplification: The current model evaluates only a single orbital plane. Real-world mega-constellations feature multiple intersecting orbital planes, which will significantly complicate the satellite handover process.
- Edge Effects: The authors noted "edge effects" (performance fluctuations) toward the end of simulation episodes due to the finite time-horizon setup.
- Scalability to Multi-Link Scenarios: A single-agent DQN introduces high overhead if scaled to multiple ground terminals. The authors point to Multi-Agent Reinforcement Learning (MARL) and distributed Actor-Critic architectures as essential next steps to make this approach practical for dense, real-world networks.
Summary updated