Imitation Learning Backoff: Reinforcement Learning-based Channel Access for Guaranteeing Fairness (Student Abstract)

Taegyeom Lee, Ohyun JoAAAI 2025 · DOI

Abstract

In multi-agent Unmanned Aerial Vehicle (UAV) networks, standard reinforcement learning (RL) often results in unfair channel access, where a subset of nodes monopolizes bandwidth. This paper presents Imitation Learning Backoff (ILB), a two-stage learning framework that first imitates the fair CSMA/CA protocol before optimizing for performance. Our results show that ILB significantly improves fairness (up to 41%) and throughput compared to traditional backoff mechanisms and standard RL approaches.

Introduction

As UAV swarms become more prevalent, efficient and fair communication channel sharing is paramount. The Contention Window (CW) determines the backoff time before transmission, and optimizing this via RL has shown promise for throughput. However, the non-stationarity of multi-agent environments often leads to "selfish" behavior, where some nodes learn to dominate the channel at the expense of others. This "winner-takes-all" scenario disrupts network reliability and mission success.

Methodology: Imitation Learning Backoff (ILB)

The proposed ILB model addresses non-stationarity through a two-stage distributed learning process using a Deep Q-Network (DQN).

Two-Stage Learning

  1. Policy Imitation: Agents initially imitate the CSMA/CA protocol, which serves as a fair baseline expert. This phase synchronizes the behavior of all UAVs, establishing a "social contract" of fairness.
  2. Weight Adjustment (WA): Once the exploration parameter $\epsilon$ reaches its minimum, the agents transition to optimizing their neural network weights for performance. This allows the network to exceed CSMA/CA's efficiency without sacrificing the established fairness.
Overview of the ILB framework, illustrating the transition from policy imitation to weight adjustment within the DQN training loop.

Technical Implementation

The DQN agents observe a state space including collision ratios, transmission delays, and Received Signal Strength (RSS). The reward function is bifurcated: $r_d$ penalizes increased delay to promote fairness, while $r_c$ rewards successful transmissions and penalizes collisions to drive throughput.

Results and Discussion

The ILB model was evaluated in simulated environments with 3, 4, and 5 UAVs. Performance was benchmarked against standard CSMA/CA and a baseline RL model, Learning Backoff (LB).

Comparative performance metrics (Fairness F, Throughput T, and Delay D) for CSMA/CA, LB, ILB, and ILB+WA across different UAV swarm sizes.

Fairness and Throughput

As shown in the results table, the standard RL model (LB) suffers from poor fairness (F = 0.70 in a 5-UAV scenario). In contrast, ILB maintains near-perfect fairness (F = 0.99) across all tested scenarios. Furthermore, the ILB+WA variant achieves superior throughput (T = 0.96 for 3 UAVs) compared to the 0.89 achieved by CSMA/CA.

Latency Trade-offs

While ILB introduces a marginal increase in delay compared to the basic CSMA/CA protocol, it remains significantly more stable than the LB model. In LB, "blocked" nodes experience extreme delays, whereas ILB ensures consistent access for all participants.

Conclusion

Imitation Learning Backoff (ILB) successfully mitigates the non-stationarity problem in multi-agent UAV networks. By leveraging imitation learning as a stabilizer, the framework achieves a high-performance "sweet spot" between throughput and fairness. Future research will focus on the scalability of ILB in larger swarms and the impact of different expert policies on learning efficiency.

Summary updated

BibTeX
@inproceedings{lee2025imitation,
  author = {Taegyeom Lee and Ohyun Jo},
  title = {{Imitation Learning Backoff: Reinforcement Learning-based Channel Access for Guaranteeing Fairness (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35266},
}
Figure