Deep Reinforcement Learning Aided Packet-Routing for Aeronautical Ad-Hoc Networks Formed by Passenger Planes

Dong Liu, Jingjing Cui, Jian-Kang Zhang, Chenyang Yang, Lajos Hanzo

TL;DR

This paper tackles the challenge of efficient data routing in fast-changing aeronautical ad-hoc networks (AANETs) by employing deep reinforcement learning. They propose two novel DRL-based routing algorithms, DQN-routing and DVN-routing, which learn optimal routing policies from historical flight data to minimize end-to-end delay, with DVN-routing showing near-optimal performance by incorporating real-time feedback.

What Problem Does This Paper Address?

Connecting passengers on airplanes to the internet is a growing need, but traditional satellite or direct air-to-ground (A2G) solutions often suffer from high latency or limited coverage. Aeronautical ad-hoc networks (AANETs), formed by airplanes acting as relays, offer a promising alternative. However, the extremely dynamic topology of these networks, due to the high speed of aircraft, makes traditional routing protocols that rely on maintaining routing tables inefficient and prone to significant overhead and latency. Existing position-based routing protocols, while faster, can get stuck in "communication voids" where no neighbor is closer to the destination. This paper aims to develop a routing strategy that can infer global network topology from local observations to overcome these limitations and minimize data transmission delay in AANETs.

Methods

The core of the paper lies in applying Deep Reinforcement Learning (DRL) to the routing problem.

  • DQN-Routing: This approach uses a Deep Q-Network (DQN) to learn a direct mapping from local geographic information (positions of the current node, its neighbors, and the destination) to the optimal routing decision (which neighbor to forward the packet to). The DQN is trained offline using historical flight data, allowing each airplane to store and use this learned policy during flight without requiring real-time network updates.
  • DVN-Routing with Feedback: To improve learning efficiency and online adaptability, this method introduces a Deep Value Network (DVN). The DVN learns a value function that estimates the expected future delay. Crucially, it incorporates a feedback mechanism where potential next-hop neighbors report their real-time queuing delay. This allows the forwarding node to "plan one step ahead," making more informed decisions and effectively bypassing congested nodes.

Both methods frame the routing problem as an RL task where the "agent" moves with the packet, and the goal is to minimize the cumulative reward (which represents delay).

Datasets

The paper does not use publicly available real-world flight data. Instead, they generate synthetic flight data for training and testing. This synthetic data mimics the mobility of airplanes within a defined 3D airspace, including preplanned flight paths and random deviations to simulate real-world uncertainties in flight positions. They also introduce "no-fly zones" to create communication voids and evaluate the routing algorithms' robustness. The dataset consists of 2000 snapshots, with half used for training and the other half for testing.

Key Insights and Ideas

  • Learning Global Topology from Local Views: The most significant insight is that DRL can learn to infer the global network topology and make optimal routing decisions by observing only local geographic information. This bypasses the need for explicit global state maintenance, which is infeasible in highly dynamic AANETs.
  • Offline Training for Scalability: Training the routing policy offline on historical data is a key enabler for deploying these intelligent routing capabilities on individual aircraft without requiring constant online updates or extensive communication overhead.
  • Feedback for Real-time Adaptability: The DVN-routing's feedback mechanism is a clever way to inject real-time network conditions (specifically, queuing delay) into an otherwise offline-trained policy, significantly improving its performance in dynamic scenarios.
  • DVN as a More Efficient Alternative: The DVN approach, by learning a value function rather than an action-value function, requires fewer parameters than DQN, leading to potentially faster training and better generalization.

Key Results

  • Reduced End-to-End (E2E) Delay: Both DQN-routing and DVN-routing consistently achieve lower E2E delay compared to the benchmark GPSR protocol.
  • Near-Optimal Performance: DVN-routing demonstrates performance very close to an "optimal" routing strategy that assumes perfect global information, significantly outperforming DQN-routing, especially when queuing delays are considered.
  • Robustness to Communication Voids: The paper shows that the DRL-based approaches are better at navigating communication voids compared to GPSR, which struggles in such scenarios.
  • Learning Curves: Simulation results show that both DQN and DVN achieve significant performance improvements after approximately 500 training episodes.

Limitations and Future Work

  • Synthetic Data: The reliance on synthetic data means the performance might differ when applied to real-world flight data, which could have more complex and unpredictable patterns.
  • Queuing Delay Assumption: While DVN-routing incorporates real-time queuing delay, the initial offline training for both DQN and DVN assumes constant queuing delay. The paper notes that this assumption is simplified for training purposes.
  • Low-Density Regions: The authors acknowledge that AANETs might struggle in regions with very low flight density.
  • Future Work: The paper suggests integrating low Earth orbit (LEO) satellites into AANETs to achieve truly global coverage, addressing the limitations of network connectivity in sparse areas.

Summary updated

Abstract

Data packet routing in aeronautical ad-hoc networks (AANETs) is challenging due to their high-dynamic topology. In this paper, we invoke deep reinforcement learning for routing in AANETs aiming at minimizing the end-to-end (E2E) delay. Specifically, a deep Q-network (DQN) is conceived for capturing the relationship between the optimal routing decision and the local geographic information observed by the forwarding node. The DQN is trained in an offline manner based on historical flight data and then stored by each airplane for assisting their routing decisions during flight. To boost the learning efficiency and the online adaptability of the proposed DQN-routing, we further exploit the knowledge concerning the system's dynamics by using a deep value network (DVN) conceived with a feedback mechanism. Our simulation results show that both DQN-routing and DVN-routing achieve lower E2E delay than the benchmark protocol, and DVN-routing performs similarly to the optimal routing that relies on perfect global information.
BibTeX
@article{liu2021deep,
  author = {Dong Liu and Jingjing Cui and Jian-Kang Zhang and Chenyang Yang and Lajos Hanzo},
  title = {{Deep Reinforcement Learning Aided Packet-Routing for Aeronautical Ad-Hoc Networks Formed by Passenger Planes}},
  journal = {IEEE Trans. Vehicular Technology},
  year = {2021},
  url = {https://openreview.net/forum?id=VgbYQkjIVM},
}
Figure