Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning

Mahmoud Assran, Joshua Romoff, Nicolas Ballas, Joelle Pineau, Mike Rabbat

Abstract

Scaling Deep Reinforcement Learning (RL) often involves parallelizing simulators to accelerate data generation. However, existing methods face a trade-off between the synchronization overhead of synchronous approaches (A2C) and the instability caused by gradient staleness in asynchronous ones (A3C). We introduce GALA (Gossip-based Actor-Learner Architectures), a decentralized framework that utilizes peer-to-peer communication to synchronize agent parameters. GALA achieves high hardware utilization and training stability at scale, outperforming standard baselines in energy efficiency and computational throughput.

Introduction

Training Deep RL agents is computationally intensive, requiring millions of environment interactions. Multi-simulator training is the standard solution, but coordination remains a bottleneck. Synchronous methods like A2C suffer from the "straggler effect," where the entire system is only as fast as the slowest simulator or CPU core. Asynchronous methods like A3C avoid this but introduce "gradient staleness," leading to instability as the number of simulators grows. GALA provides a middle ground: a system as fast as asynchronous methods but with the stability of synchronous ones.

Methodology: The GALA Framework

GALA replaces global synchronization with a decentralized gossip protocol. Agents are organized into a peer-to-peer network topology (e.g., a ring).

The Gossip Protocol

Each agent maintains local parameters and performs updates independently. Periodically, agents "gossip" by sending their parameters to neighbors and averaging received updates. This communication is non-blocking, allowing agents to continue processing without waiting for handshakes.

Theoretical Foundation: The ε-ball Guarantee

A key contribution of GALA is the proof that agents' policies remain within a bounded distance (an ε-ball) of each other. This ensures that the decentralized group converges toward a common solution without a central master model.

Experimental Evaluation

We evaluated GALA-A2C on six Atari-2600 games (BeamRider, Breakout, Pong, Qbert, Seaquest, and SpaceInvaders) via the Arcade Learning Environment.

Stability at Scale

A significant advantage of GALA is its robustness to increasing simulator counts. While vanilla A2C performance degrades sharply beyond 128 simulators, GALA maintains high convergence rates even at 1024 simulators.

Percentage of convergent runs as the number of simulators increases across different Atari games.

Computational and Energy Efficiency

GALA improves GPU utilization by up to 10x compared to A2C by reducing idle time. This translates to higher throughput and superior energy efficiency, producing significantly more frames per kilojoule than standard methods.

Reward vs. training time (hrs) for GALA-a2c compared to A2C, A3C, and IMPALA.

Key Insights

  • Beneficial Diversity: The slight policy differences between GALA agents act as a form of beneficial noise that decorrelates gradients, preventing the training collapse often seen in large-scale synchronous training.
  • On-Policy Performance: GALA-A2C achieves competitive sample efficiency using default hyperparameters, often outperforming more complex off-policy methods like IMPALA that require importance sampling corrections.

Conclusion and Future Work

GALA demonstrates that decentralized communication is a viable and efficient alternative for scaling Deep RL. Future research will explore more complex network topologies, such as expander graphs, and the extension of gossip architectures to off-policy methods utilizing replay buffers.

Summary updated

BibTeX
@inproceedings{assran2019gossip,
  author = {Mahmoud Assran and Joshua Romoff and Nicolas Ballas and Joelle Pineau and Mike Rabbat},
  title = {{Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=YzxbWkjPlE4},
}
Figure