FedRDR: Federated Reinforcement Distillation-Based Routing Algorithm in UAV-Assisted Networks for Communication Infrastructure Failures

Jie Li, Anqi Liu, Guangjie Han, Shuang Cao, Feng Wang, Xingwei WangDrones 2024 · DOI

TL;DR

This paper introduces FedRDR, a novel routing algorithm for UAV-assisted emergency networks that leverages federated reinforcement distillation. FedRDR significantly reduces communication overhead and accelerates training by distilling knowledge from experienced agents, making it ideal for resource-constrained disaster scenarios.

What Problem Does This Paper Address?

Traditional Internet of Things (IoT) networks often fail during natural disasters due to damaged infrastructure. This leaves emergency responders without crucial communication capabilities. While Unmanned Aerial Vehicles (UAVs) offer a promising solution for establishing temporary communication networks, efficiently routing data in these dynamic, resource-constrained environments is a significant challenge. Existing approaches struggle with limited data per domain, slow convergence, and high communication costs associated with federated learning. This paper aims to address these issues by developing a routing algorithm that is efficient, privacy-preserving, and adaptable to the volatile conditions of disaster-stricken areas.

Methods

The core of this work is the FedRDR (Federated Reinforcement Distillation-Based Routing) algorithm. It builds upon a hierarchical, multi-domain architecture inspired by Software-Defined Networking (SDN), where high-performance UAVs act as domain controllers.

Key methodological components include:

  • Hierarchical Multi-Domain Architecture: This architecture uses a global server and multiple UAV-based domain controllers to manage network resources and facilitate communication.
  • Federated Reinforcement Learning (FRL): This approach allows UAVs to train routing models locally without sharing raw data, preserving privacy. The server aggregates model updates from multiple domains to create a global model.
  • Knowledge Distillation (KD): Instead of transmitting raw model parameters, FedRDR uses knowledge distillation. This involves transferring "proxy experience memory" (states, actions, and rewards) from experienced agents to newer ones. This significantly reduces the amount of data that needs to be transmitted.
  • Deep Q-Network (DDQN): The routing decision module within each domain controller utilizes a DDQN to learn optimal routing paths based on network traffic and link delay.
  • Proxy Experience Memory: This novel concept allows agents to share distilled knowledge in a privacy-preserving manner, enhancing generalization and accelerating learning.

Datasets

The paper does not mention specific real-world datasets. Instead, it relies on a simulated environment for evaluation.

  • Simulation Environment: A Mininet network simulator with a Ryu controller was used to create a simulated drone network. The experiments were implemented in Python with the PyTorch framework.

Key Insights and Ideas

  • Distillation is Key for Efficiency: The core insight is that distilling knowledge (proxy experience memory) is far more efficient than transmitting raw model parameters in federated learning for UAV networks. This dramatically reduces communication overhead.
  • Accelerated Training in Dynamic Environments: FedRDR significantly speeds up the training process for intelligent agents within individual domains. This is crucial for rapidly adapting to changing network conditions during emergencies.
  • Privacy Preservation with Enhanced Generalization: By using federated learning and proxy experience memory, the algorithm maintains user data privacy while improving the generalization capabilities of the routing models across different domains.
  • UAVs as Intelligent Network Controllers: The paper highlights the potential of using high-performance UAVs not just as communication relays but as intelligent domain controllers capable of sophisticated routing decisions.

Key Results

  • Reduced Communication Overhead: FedRDR reduces the average communication data size between domain controllers and the server to approximately 45.3 KB, a reduction of about 29% compared to traditional federated reinforcement learning algorithms.
  • Faster Convergence: The FedRDR algorithm demonstrates faster convergence compared to standard federated reinforcement learning and other baseline routing algorithms (DFS, Q-learning). For instance, in one experiment, FedRDR achieved convergence in about 40 iterations, while federated reinforcement learning took around 60 iterations.
  • Improved Routing Performance: In simulations with increased network complexity (8 switches and multiple flows), FedRDR showed superior performance in terms of network delay compared to DFS and Q-learning, which struggled to converge or maintain optimal routing.
  • Energy Efficiency: By accelerating training, FedRDR can lead to reduced energy consumption for UAV domain controllers.

Limitations and Future Work

  • Simulation-Based Evaluation: The study's findings are based on a simulated environment. While the simulation is described as robust, real-world deployment may introduce additional complexities and challenges.
  • Scalability with More Domains: While the paper shows that increasing the number of domains doesn't negatively impact accuracy, it does increase the amount of uploaded data. Further research could explore more advanced aggregation techniques for even larger-scale deployments.
  • Adaptability to Diverse Disaster Scenarios: The current work focuses on general communication infrastructure failures. Future work could investigate FedRDR's performance in highly specific disaster scenarios with unique network constraints.
  • Real-World Deployment and Testing: The authors suggest that the proposed architecture and algorithm are universal and can be adapted to more complex networks, implying that real-world testing and validation would be a logical next step.

Summary updated

Abstract

Traditional Internet of Things (IoT) networks have limited coverage and may experience failures due to natural disasters affecting critical IoT devices, making it difficult for them to provide communication services. Therefore, how to establish network communication service more efficiently in the presence of fault points is the problem we solve in this paper. To address this issue, this study constructs a hierarchical multi-domain data transmission architecture for an emergency network with unmanned aerial vehicles (UAVs) employed as core communication devices. This architecture expands the functionality of UAVs as key network devices and provides a theoretical basis for their feasibility as intelligent network controllers and switches. Firstly, the UAV controllers perceive the network status and learn the spatio-temporal characteristics of air-to-ground network links. Secondly, a routing algorithm within the domain based on federated reinforcement distillation (FedRDR) is developed, which enhances the generalization capability of the routing decision model by increasing the training data samples. Simulation experiments are conducted, and the results show that the average communication data size between each domain controller and the server is approximately 45.3 KB when using the FedRDR algorithm. Compared to the transmission of parameters through federated reinforcement learning algorithms, FedRDR reduces the transmitted parameter size by approximately 29%. Therefore, the FedRDR routing algorithm helps to facilitate knowledge transfer, accelerate the training process of intelligent agents within the domain, and reduce communication costs in resource-constrained scenarios for UAV networks and has practical value.
BibTeX
@inproceedings{li2024fedrdr,
  author = {Jie Li and Anqi Liu and Guangjie Han and Shuang Cao and Feng Wang and Xingwei Wang},
  title = {{FedRDR: Federated Reinforcement Distillation-Based Routing Algorithm in UAV-Assisted Networks for Communication Infrastructure Failures}},
  booktitle = {Drones},
  year = {2024},
  doi = {10.3390/drones8020049},
}
Figure