Abstract
Unmanned aerial vehicles (UAVs) are capable of serving as aerial base stations (BSs) for providing both cost-effective and on-demand wireless communications. This article investigates dynamic resource allocation of multiple UAVs enabled communication networks with the goal of maximizing long-term rewards. More particularly, each UAV communicates with a ground user by automatically selecting its communicating users, power levels and subchannels without any information exchange among UAVs. To model the uncertainty of environments, we formulate the long-term resource allocation problem as a stochastic game for maximizing the expected rewards, where each UAV becomes a learning agent and each resource allocation solution corresponds to an action taken by the UAVs. Afterwards, we develop a multi-agent reinforcement learning (MARL) framework that each agent discovers its best strategy according to its local observations using learning. More specifically, we propose an agent-independent method, for which all agents conduct a decision algorithm independently but share a common structure based on Q-learning. Finally, simulation results reveal that: 1) appropriate parameters for exploitation and exploration are capable of enhancing the performance of the proposed MARL based resource allocation algorithm; 2) the proposed MARL algorithm provides acceptable performance compared to the case with complete information exchanges among UAVs. By doing so, it strikes a good tradeoff between performance gains and information exchange overheads.
Introduction
The deployment of Unmanned Aerial Vehicles (UAVs) as aerial base stations offers a flexible solution for enhancing wireless network capacity and coverage. However, the high mobility of UAVs and the dynamic nature of the wireless environment present significant challenges for resource management. Traditional centralized optimization techniques often struggle with scalability and the high communication overhead required for global state information. This paper addresses the problem of dynamic, decentralized resource allocation where multiple UAVs must independently decide on user association, subchannel selection, and power levels to maximize long-term network efficiency.
System Model and Problem Formulation
The network is modeled as a stochastic game where multiple UAVs operate within a defined area to serve ground users. Each UAV acts as an independent agent. The goal is to maximize a long-term reward function that balances data throughput against power consumption, subject to Quality of Service (QoS) constraints defined by Signal-to-Interference-plus-Noise Ratio (SINR) thresholds.
As illustrated in the system model, each UAV follows a trajectory and must make resource allocation decisions at each time slot. The action space for each agent is a triplet consisting of a target user, a specific subchannel, and a transmission power level. Because UAVs do not exchange information, the environment is non-stationary from the perspective of an individual agent, as the interference patterns change based on the actions of other UAVs.
Multi-Agent Reinforcement Learning Framework
To solve this decentralized problem, a multi-agent reinforcement learning (MARL) framework based on independent Q-learning is proposed. Each UAV maintains its own Q-table to estimate the expected long-term rewards for different state-action pairs.
Independent Learners (ILs)
By treating other UAVs as part of the stochastic environment, the complexity of the coordination problem is significantly reduced. This "agent-independent" approach eliminates the need for a backhaul network for UAV-to-UAV communication, making it highly suitable for real-time applications.
Exploration vs. Exploitation
The agents employ an $\epsilon$-greedy strategy. Simulations indicate that a balanced exploration rate (e.g., $\epsilon = 0.5$) is crucial in these dynamic environments to allow agents to adapt to the changing network geometry caused by UAV mobility.
Performance Evaluation
The proposed MARL algorithm was evaluated in a simulated environment with 100 to 200 ground users and 2 to 4 UAVs.
- Convergence: The algorithm demonstrates stable convergence, with average rewards plateauing after approximately 250 time slots.
- Efficiency: While centralized methods like "Matching Theory" (which assume perfect information) provide a performance upper bound, the MARL approach achieves comparable results with zero communication overhead.
- Mobility: The study confirms that UAV speed inversely affects long-term rewards, as higher speeds reduce the effective service time for specific user clusters.
Conclusion
This research demonstrates that decentralized MARL is a viable and efficient approach for resource allocation in UAV-aided networks. By allowing UAVs to learn optimal strategies independently, the system achieves a favorable balance between throughput performance and signaling overhead. Future work will explore the joint optimization of UAV trajectories and resource allocation, as well as the impact of variable altitudes on network performance.
Summary updated