Abstract
The paper considers independent reinforcement learning (IRL) for multi-agent decision-making processes in the paradigm of federated learning (FL). We show that FL can clearly improve the policy performance of IRL in terms of training efficiency and stability. However, since the policy parameters are trained locally and aggregated iteratively through a central server in FL, frequent information exchange incurs a large amount of communication overheads. To reach a good balance between improving the model's convergence performance and reducing the required communication and computation overheads, this paper proposes a system utility function and develops a consensus-based optimization scheme on top of the periodic averaging method, which introduces the consensus algorithm into FL for the exchange of a model's local gradients. This paper also provides novel convergence guarantees for the developed method, and demonstrates its superior effectiveness and efficiency in improving the system utility value through theoretical analyses and numerical simulation results.
Introduction
In multi-agent systems such as autonomous traffic control or drone swarms, agents often utilize Independent Reinforcement Learning (IRL) to develop individual policies. While simple to implement, IRL frequently suffers from instability because the environment is non-stationary—it changes constantly as other agents learn and adapt. Federated Learning (FL) offers a solution by having a central server aggregate local models from all agents, effectively "averaging" their experiences to create a more robust global policy.
However, a "naive" implementation of FL requires agents to transmit massive amounts of data to a central server with high frequency. In wireless edge networks, this leads to significant communication overheads, battery depletion, and latency. The core challenge addressed in this research is finding the "sweet spot": reducing communication frequency without compromising the model's convergence or performance.
The FIRL Framework
The authors propose a framework called Federated IRL (FIRL) enhanced with two key strategies to balance performance and overhead:
- Periodic Averaging: Instead of syncing with the central server at every training step, agents perform several local updates (denoted by $\tau$) before transmitting their gradients to the cloud.
- Consensus Mechanism: This is the paper's primary innovation. Before performing a local update, agents exchange their gradients with immediate neighbors via Device-to-Device (D2D) communication. This "local gossip" helps agents reach a consensus on the gradient direction, reducing the variance of their updates and mimicking the benefits of a global sync without the cost of cloud communication.
Theoretically, the authors prove convergence guarantees by linking the model's error bound to the algebraic connectivity of the agent network. Essentially, the better connected the agents are locally, the faster and more reliably the system converges.
Experimental Evaluation
The researchers evaluated their method using the "Figure Eight" simulation scenario, a standard benchmark for mixed-autonomy traffic control. The environment consists of a one-way lane shaped like an "8" where 14 vehicles must navigate an intersection without crashing. In this setup, half the vehicles are controlled by Deep Reinforcement Learning (DRL) agents using the Proximal Policy Optimization (PPO) algorithm, while the other half follow standard traffic models (SUMO). The primary goal is to maximize the Normalized Average Speed (NAS) of the entire fleet.
Key Results and Insights
Superior Stability and Performance
In the Figure Eight task, the Federated IRL (FIRL) approach significantly outperformed naive IRL. As shown in the training curves, FIRL achieves higher average speeds with much greater stability than the highly volatile IRL baseline.
Efficiency Gains
The consensus-based method (FIRL_C) achieved an Expected Gradient Norm of ~1.55 (where lower indicates better convergence). This performance is comparable to syncing with the server every single step, but it was achieved with 10x fewer central server communications. By adjusting the number of local updates ($\tau$) and local interactions ($E$), the system maintained high traffic speeds while dramatically slashing total communication costs.
Local Gossip as a Regularizer
By exchanging gradients with neighbors, agents effectively smooth out the "noise" in their local reinforcement learning process. The paper highlights that the physical layout of the agent network directly impacts learning; a more densely connected local network allows for a larger "step size" in the consensus phase, leading to lower error bounds.
Conclusion
This research demonstrates that consensus-based D2D communication can serve as a powerful, low-cost alternative to frequent global synchronization in Federated Reinforcement Learning. While the current model assumes a strongly connected, undirected graph and omits potential signal interference, it provides a robust foundation for future work in heterogeneous multi-agent systems and hierarchical scaling for large-scale edge deployments.
Summary updated