Dynamic Trajectory and Power Control in Ultra-Dense UAV Networks: A Mean-Field Reinforcement Learning Approach

Fei Song, Zhe Wang, Jun Li, Long Shi, Wen Chen, Shi Jin

TL;DR

This paper tackles the challenge of optimizing resource allocation and trajectory for a large number of UAVs serving ground users with dynamic demands. They introduce a novel "Maximum Entropy Mean-Field Deep Q Network" (ME-MFDQN) algorithm that uses mean-field game theory and reinforcement learning to efficiently manage these complex, large-scale networks, leading to improved energy efficiency.

What Problem Does This Paper Address?

In modern wireless communication, Unmanned Aerial Vehicles (UAVs) are increasingly being deployed as aerial base stations to enhance coverage and provide flexible services. However, when these UAVs become very numerous (ultra-dense networks) and serve users whose demands change unpredictably over time, coordinating their actions becomes incredibly difficult. Each UAV needs to decide where to fly, which users to serve, and how much power to use, all while dealing with interference from other UAVs and limited energy. Traditional methods struggle with the sheer scale and dynamic nature of these problems. This paper aims to solve this by developing a learning-based approach that can handle a massive number of UAVs and time-varying user demands efficiently.

Methods

The core of the paper's approach lies in two key concepts: Mean-Field Games (MFGs) and Deep Reinforcement Learning (DRL).

  • Mean-Field Games (MFGs): For a very large number of interacting agents (UAVs in this case), directly modeling every interaction is computationally infeasible. MFGs simplify this by treating the collective behavior of all other agents as a single "mean-field." This reduces the problem from a multi-agent game to a two-player game between a representative agent (one UAV) and this mean-field. The authors prove that a unique "mean-field equilibrium" exists for this setup.
  • Maximum Entropy Mean-Field Deep Q Network (ME-MFDQN): To find this equilibrium in a practical, model-free way (meaning they don't need to know the exact system dynamics beforehand), they propose ME-MFDQN. This algorithm builds upon Deep Q-Networks (DQNs) but incorporates two crucial enhancements:
  • Mean-Field Integration: It uses the mean-field distribution to inform the decision-making of the representative UAV.
  • Maximum Entropy: By adding an entropy term to the objective function, the algorithm encourages exploration of different actions, preventing it from getting stuck in suboptimal solutions and improving its ability to find the true mean-field equilibrium.
  • Partially Observable Extension: The algorithm is also extended to handle scenarios where UAVs have limited observation capabilities of their surroundings.

The solution involves an iterative two-step process: first, the representative UAV learns its best response policy given the current mean-field; second, the mean-field distribution is updated based on this learned policy. This process repeats until convergence.

Datasets

Not discussed in the paper. The paper focuses on a simulated environment with parameters derived from existing literature for channel modeling and UAV propulsion.

Key Insights and Ideas

  • Scalability through Mean-Field: The paper demonstrates that MFGs are a powerful tool for abstracting complex interactions in large-scale multi-agent systems, making them tractable for optimization.
  • Exploration is Key: Incorporating maximum entropy into the reinforcement learning objective is crucial for escaping local optima and finding robust solutions in dynamic environments.
  • Dynamic Demands Matter: The performance of the UAVs is significantly influenced by how predictable user demands are. More predictable demands (higher self-transition probability) lead to better energy efficiency.
  • Observation Radius Impacts Strategy: Limited observation forces UAVs to adopt more conservative strategies (e.g., lower transmit power) or compensate with more flying, impacting overall efficiency.

Key Results

  • Improved Energy Efficiency: The proposed ME-MFDQN algorithm consistently outperforms benchmark algorithms (e-greedy IDQN, e-greedy MFDQN, Boltzmann MFDQN) in terms of average reward and energy efficiency.
  • Robustness to Network Size: The trained policies show robustness to slight variations in the number of UAVs, suggesting good generalization capabilities.
  • Impact of User Demand Correlation: Higher temporal correlation in GUs' service demands leads to increased energy efficiency, as UAVs can better predict and adapt their strategies.
  • Impact of Observation: Wider observation radius for UAVs significantly improves energy efficiency, as they can make more informed decisions and reduce unnecessary flying. For instance, 75% observability showed better results than 25% observability.

Limitations and Future Work

  • Homogeneous UAVs: The current formulation assumes all UAVs are homogeneous and indistinguishable. Future work could explore scenarios with heterogeneous UAV capabilities.
  • Simplified User Model: The paper uses a simplified two-state Markov process for user demands. More complex and realistic demand models could be investigated.
  • Computational Complexity: While MFGs reduce complexity, the training of deep reinforcement learning models can still be computationally intensive.
  • Real-world Deployment: The results are based on simulations. Future work could involve testing the algorithm in real-world UAV deployments to validate its practical performance.
  • Extension to Other Scenarios: The authors suggest extending the approach to other scenarios like cooperative UAV networks or different communication paradigms.

Summary updated

Abstract

In ultra-dense unmanned aerial vehicle (UAV) networks, it is challenging to coordinate the resource allocation and interference management among large-scale UAVs, for providing flexible and efficient service coverage to the ground users (GUs). In this paper, we propose a learning-based resource allocation scheme in an ultra-dense UAV communication network, where the GUs' service demands are time-varying with unknown distributions. We formulate the non-cooperative game among multiple co-channel UAVs as a stochastic game, where each UAV jointly optimizes its trajectory, user association, and downlink power control to maximize the expectation of its locally cumulative energy efficiency under the interference and energy constraints. To cope with the scalability issue in a large-scale network, we further formulate the problem as a mean-field game (MFG), which simplifies the interactions among the UAVs into a two-player game between a representative UAV and a mean-field. We prove the existence and uniqueness of the equilibrium for the MFG, and propose a model-free mean-field reinforcement learning algorithm named maximum entropy mean-field deep Q network (ME-MFDQN) to solve the mean-field equilibrium in both fully and partially observable scenarios. The simulation results reveal that the proposed algorithm improves the energy efficiency compared with the benchmark algorithms. Moreover, the performance can be further enhanced if the GUs' service demands exhibit higher temporal correlation or if the UAVs have wider observation capabilities over their nearby GUs.
BibTeX
@article{song2025dynamic,
  author = {Fei Song and Zhe Wang and Jun Li and Long Shi and Wen Chen and Shi Jin},
  title = {{Dynamic Trajectory and Power Control in Ultra-Dense UAV Networks: A Mean-Field Reinforcement Learning Approach}},
  journal = {IEEE Trans. Wireless Communications},
  year = {2025},
  url = {https://openreview.net/forum?id=MmNkbkpzT2},
}
Figure