Abstract
In multi-agent Unmanned Aerial Vehicle (UAV) networks, standard reinforcement learning (RL) often results in unfair channel access, where a subset of nodes monopolizes bandwidth. This paper presents Imitation Learning Backoff (ILB), a two-stage learning framework that first imitates the fair CSMA/CA protocol before optimizing for performance. Our results show that ILB significantly improves fairness (up to 41%) and throughput compared to traditional backoff mechanisms and standard RL approaches.
Introduction
As UAV swarms become more prevalent, efficient and fair communication channel sharing is paramount. The Contention Window (CW) determines the backoff time before transmission, and optimizing this via RL has shown promise for throughput. However, the non-stationarity of multi-agent environments often leads to "selfish" behavior, where some nodes learn to dominate the channel at the expense of others. This "winner-takes-all" scenario disrupts network reliability and mission success.
Methodology: Imitation Learning Backoff (ILB)
The proposed ILB model addresses non-stationarity through a two-stage distributed learning process using a Deep Q-Network (DQN).
Two-Stage Learning
- Policy Imitation: Agents initially imitate the CSMA/CA protocol, which serves as a fair baseline expert. This phase synchronizes the behavior of all UAVs, establishing a "social contract" of fairness.
- Weight Adjustment (WA): Once the exploration parameter $\epsilon$ reaches its minimum, the agents transition to optimizing their neural network weights for performance. This allows the network to exceed CSMA/CA's efficiency without sacrificing the established fairness.
Technical Implementation
The DQN agents observe a state space including collision ratios, transmission delays, and Received Signal Strength (RSS). The reward function is bifurcated: $r_d$ penalizes increased delay to promote fairness, while $r_c$ rewards successful transmissions and penalizes collisions to drive throughput.
Results and Discussion
The ILB model was evaluated in simulated environments with 3, 4, and 5 UAVs. Performance was benchmarked against standard CSMA/CA and a baseline RL model, Learning Backoff (LB).
Fairness and Throughput
As shown in the results table, the standard RL model (LB) suffers from poor fairness (F = 0.70 in a 5-UAV scenario). In contrast, ILB maintains near-perfect fairness (F = 0.99) across all tested scenarios. Furthermore, the ILB+WA variant achieves superior throughput (T = 0.96 for 3 UAVs) compared to the 0.89 achieved by CSMA/CA.
Latency Trade-offs
While ILB introduces a marginal increase in delay compared to the basic CSMA/CA protocol, it remains significantly more stable than the LB model. In LB, "blocked" nodes experience extreme delays, whereas ILB ensures consistent access for all participants.
Conclusion
Imitation Learning Backoff (ILB) successfully mitigates the non-stationarity problem in multi-agent UAV networks. By leveraging imitation learning as a stabilizer, the framework achieves a high-performance "sweet spot" between throughput and fairness. Future research will focus on the scalability of ILB in larger swarms and the impact of different expert policies on learning efficiency.
Summary updated