Abstract: Unmanned aerial vehicles (UAVs) are capable of serving as aerial base stations (BSs) for providing both cost-effective and on-demand wireless communications. This article investigates dynamic resource allocation of multiple UAVs enabled communication networks with the goal of maximizing long-term rewards. More particularly, each UAV communicates with a ground user by automatically selecting its communicating user, power level and subchannel without any information exchange among UAVs. To model the dynamics and uncertainty in environments, we formulate the long-term resource allocation problem as a stochastic game for maximizing the expected rewards, where each UAV becomes a learning agent and each resource allocation solution corresponds to an action taken by the UAVs. Afterwards, we develop a multi-agent reinforcement learning (MARL) framework that each agent discovers its best strategy according to its local observations using learning. More specifically, we propose an agent-independent method, for which all agents conduct a decision algorithm independently but share a common structure based on Q-learning. Finally, simulation results reveal that: 1) appropriate parameters for exploitation and exploration are capable of enhancing the performance of the proposed MARL based resource allocation algorithm; 2) the proposed MARL algorithm provides acceptable performance compared to the case with complete information exchanges among UAVs. By doing so, it strikes a good tradeoff between performance gains and information exchange overheads.
Introduction
Unmanned Aerial Vehicles (UAVs) are increasingly deployed as "flying base stations" to provide on-demand wireless coverage during emergencies or at crowded events. However, managing a fleet of these aerial platforms presents significant logistical challenges. When multiple UAVs operate in the same vicinity, they compete for limited resources, including radio spectrum and power. Without coordination, simultaneous transmissions on identical frequencies can lead to severe interference, degrading the quality of service (QoS) for ground users.
Traditional resource management solutions often rely on a centralized controller with global knowledge of the network state. However, in dynamic UAV environments, the communication overhead required to maintain such centralized control is often prohibitive. This research proposes a decentralized framework where UAVs independently manage their resources using Multi-Agent Reinforcement Learning (MARL), allowing each vehicle to make local decisions that optimize the overall network performance.
Methodology: The MARL Framework
The network is modeled as a stochastic game where each UAV acts as an independent learning agent. The goal is for each agent to observe its local environment and select an action—specifically, a combination of a target user, a subchannel, and a power level—to maximize long-term energy efficiency.
Independent Q-Learning
Rather than employing a complex joint-learning system that requires inter-agent communication, the proposed framework utilizes Independent Learners (ILs). Each UAV runs its own Q-learning algorithm, treating other UAVs as part of the dynamic, stochastic environment. This significantly reduces the computational and communication burden on the network.
Reward and Exploration
The reward function is designed to maximize energy efficiency, defined as throughput minus power costs. To ensure reliability, the reward drops to zero if the Signal-to-Interference-plus-Noise Ratio (SINR) falls below a required threshold, effectively enforcing QoS as a hard constraint. To balance the discovery of new strategies with the use of known successful ones, an $\epsilon$-greedy exploration strategy is employed. Simulations indicated that an exploration rate of $\epsilon = 0.5$ provides an optimal balance for convergence.
Simulation and Results
The system was evaluated using numerical simulations based on 3GPP LTE-A standards. The environment consisted of 100 to 200 ground users randomly distributed within a 500m radius, served by 2 to 4 UAVs flying at a fixed altitude of 100m.
Key findings include:
- Performance vs. Overhead: The MARL algorithm achieved performance comparable to centralized "Matching Theory" (Gale-Shapley) algorithms. While centralized methods offer slightly higher gains, they require complete information exchange. The MARL approach provides a robust alternative with zero communication overhead.
- Convergence and Stability: The independent learners successfully converged to stable Q-values, demonstrating that decentralized agents can reach an equilibrium in complex multi-UAV settings.
- Impact of Mobility: As UAV speed increases, the average reward tends to decrease because the vehicles spend less time in optimal service positions, highlighting the critical link between trajectory and resource efficiency.
Conclusion and Future Work
This research demonstrates that decentralized intelligence via MARL is a viable path for managing complex UAV networks. By treating resource allocation as a learning problem, UAVs can achieve high performance without the overhead of centralized coordination. Future work will explore integrating trajectory control directly into the MARL agents and utilizing Deep Q-Networks (DQN) to handle larger, more complex state spaces involving more users and channels.
Summary updated