Federated Twin Delayed Deep Deterministic Policy Gradient for Delay and Energy Consumption Optimization in Urban Air Mobility with UAV-Assisted MEC

Chunyu Pan, Zhonghao Luo, Jiuchuan Zhang, Lei Shi, Jirong Yi, Zhaohui YangDrones 2025 · DOI

TL;DR

This paper tackles the critical challenge of optimizing both communication delay and energy consumption for Urban Air Mobility (UAM) systems, especially during emergencies when edge servers might fail. They propose a novel Federated Twin Delayed Deep Deterministic Policy Gradient (UF-TD3) algorithm that leverages federated learning to enable multiple UAM systems to collaboratively learn optimal resource allocation strategies, significantly outperforming existing methods.

What Problem Does This Paper Address?

Urban Air Mobility (UAM) promises to revolutionize urban transportation, but it relies heavily on real-time data transmission and processing. This creates a significant challenge: minimizing communication delay and energy consumption for UAM aircraft, especially given their limited payload and flight times. The problem is exacerbated in emergency scenarios where traditional edge computing infrastructure might be unavailable. In such situations, Unmanned Aerial Vehicles (UAVs) can act as relays and mobile edge computing (MEC) platforms to assist UAM aircraft. However, efficiently managing the complex interplay of task offloading, resource allocation, and communication across these distributed systems to simultaneously minimize both delay and energy consumption is a formidable task.

Methods

The core of this paper is the development of a novel UAM Federated Twin Delayed Deep Deterministic Policy Gradient (UF-TD3) algorithm. This approach combines:

  • Twin Delayed Deep Deterministic Policy Gradient (TD3): A state-of-the-art deep reinforcement learning algorithm known for its stability and performance in continuous action spaces, adapted for UAM resource allocation.
  • Federated Learning (FL): This allows multiple UAM systems (or their edge servers) to collaboratively train a shared model without exchanging raw data. Instead, they share model parameters, which significantly reduces communication overhead and enhances privacy. This is crucial for distributed UAM systems.
  • Hierarchical Task Scheduling and Resource Allocation: The UF-TD3 algorithm employs a strategy to distribute computational load effectively, avoiding overly complex global optimization calculations.

The algorithm involves two value networks and one policy network, with mechanisms to reduce noise in actions and control the frequency of policy and target network updates. The federated averaging process occurs at a cloud node, aggregating local models from edge servers to create a global model, which is then distributed back.

Datasets

Not discussed in the paper. The paper relies on simulated data generated based on the proposed system model and parameters.

Key Insights and Ideas

  • Federated Learning is Key for Distributed UAM: The paper strongly suggests that federated learning is essential for enabling collaborative learning and improving performance in distributed UAM systems, especially when dealing with multiple UAM aircraft and edge servers.
  • Joint Optimization is Superior: Simply optimizing for delay or energy individually is suboptimal. A joint strategy that considers a weighted sum of both factors provides a more balanced and practical solution for UAM.
  • UF-TD3's Robustness: The UF-TD3 algorithm demonstrates superior convergence speed and stability compared to other deep reinforcement learning algorithms, even in complex, non-convex optimization problems.
  • UAV Relay's Strategic Importance: The positioning of relay UAVs significantly impacts system performance, highlighting the need for intelligent deployment strategies.

Key Results

  • Significant Performance Improvement: The proposed UF-TD3 algorithm significantly outperforms four other baseline algorithms (TD3-DE, IF-DDPG-DE, DDPG-DE, DQN-DE) in optimizing delay and energy consumption.
  • Reduced Delay and Energy: Compared to conventional delay minimization strategies, the joint strategy reduces delay by 63.8%. Compared to energy minimization strategies, it reduces energy consumption by 73.96%.
  • Faster Convergence: The UF-TD3 algorithm converges much faster than baseline algorithms, reaching convergence in approximately 1500 iterations compared to over 2000 iterations for others.
  • Scalability: The UF-TD3 algorithm shows a smaller and more gradual growth rate in delay and energy consumption as the number of UAM aircraft increases, indicating good scalability.
  • Weighting Coefficient Impact: The study demonstrates how adjusting the weighting coefficient for delay and energy consumption allows for a trade-off, enabling users to prioritize either delay or energy based on specific operational requirements.

Limitations and Future Work

  • Idealized Communication Environment: The current study assumes an idealized communication environment. Future research should consider more realistic factors like interference and environmental conditions.
  • UAV Battery Capacity: While relay UAVs are assumed to be powered by ground stations, the paper acknowledges that battery capacity is a critical limiting factor for UAVs in general, though not explicitly modeled in this work.
  • Further Exploration of Weighting Coefficients: While the impact of weighting coefficients is shown, further research could explore adaptive adjustment of these coefficients based on real-time system conditions.
  • Real-World Deployment: The study is based on simulations. Future work could involve validating the proposed approach in real-world UAM scenarios.

Summary updated

Abstract

With the rapid expansion of urban populations and the accelerated pace of urbanization, the concept of urban air mobility (UAM) has emerged. During flights, UAM aircraft need to transmit real-time sensing information to base stations for further processing and analysis. Large-scale real-time data require leveraging the computing capabilities of edge servers at the network edge to reduce transmission delay and energy consumption of UAM aircraft. In cases where edge servers are unable to process information, an unmanned aerial vehicle (UAV) equipped with computing capabilities and operating in low-altitude airspace can serve as a relay to assist in communication and computation. Due to the limited payloads and flight times of UAVs and UAM aircraft, delay and energy consumption within the system pose significant challenges. To tackle these challenges, two fundamental objectives have been proposed: minimizing delay and minimizing energy consumption. Furthermore, an optimization problem has been proposed to minimize the weighted sum of delay and energy consumption. Then, a UAM federated twin delayed deep deterministic policy gradient (UF-TD3) algorithm has been proposed to solve the original problems characterized by complex, non-convex, and inseparable variables. Simulation results show that the proposed UF-TD3 algorithm converges quickly and significantly outperforms four other baseline algorithms in optimizing delay and energy consumption performance. Moreover, compared to the conventional delay minimization strategy and energy minimization strategy, the proposed strategy of minimizing the weighted sum of delay and energy consumption can reduce the delay by 63.8% and reduce energy by 73.96%.
BibTeX
@inproceedings{pan2025federated,
  author = {Chunyu Pan and Zhonghao Luo and Jiuchuan Zhang and Lei Shi and Jirong Yi and Zhaohui Yang},
  title = {{Federated Twin Delayed Deep Deterministic Policy Gradient for Delay and Energy Consumption Optimization in Urban Air Mobility with UAV-Assisted MEC}},
  booktitle = {Drones},
  year = {2025},
  doi = {10.3390/drones9020137},
}
Figure