Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning

Mahmoud Assran, Joshua Romoff, Nicolas Ballas, Joelle Pineau, Mike Rabbat

Overview

Gossip-based Actor-Learner Architectures (GALA) is a distributed framework for Deep Reinforcement Learning (RL) that replaces strict synchronization with "gossip"—a peer-to-peer communication method where agents asynchronously swap parameters with neighbors. This approach achieves the stability of synchronous methods and the speed of asynchronous ones, resulting in up to a 10-fold improvement in GPU utilization and significantly better energy efficiency.

The Challenge of Scaling RL

Scaling Deep RL is notoriously difficult. To train agents quickly, researchers usually distribute the workload across many simulators. Currently, there are two main ways to do this, both with significant trade-offs:

  1. Synchronous (e.g., A2C): All learners wait for every simulator to finish before updating. This is stable but suffers from the "straggler effect"—the entire system moves only as fast as its slowest component.
  2. Asynchronous (e.g., A3C): Learners update a shared global policy whenever they have data. This is fast but suffers from "gradient staleness"—by the time an update is applied, the policy might have changed so much that the update is no longer valid, leading to instability.

The authors sought a "middle way": a system that is robust to slow simulators (like A3C) but maintains the mathematical stability of a unified policy (like A2C).

The GALA Approach

Instead of a single master policy, GALA uses multiple independent agents organized in a peer-to-peer network (specifically a directed ring topology).

  • The Gossip Mechanism: Each agent performs its own local updates. Periodically and asynchronously, it "gossips" by sending its parameters to its neighbors and averaging its own parameters with those it receives.
  • Loosely Coupled: Because communication is non-blocking, an agent doesn't stop and wait for its neighbor to reply. It just checks its "mailbox" and moves on.
  • Theoretical Safety Net: The core contribution is a mathematical proof that even though agents update independently, their policies are guaranteed to stay within an $\epsilon$-ball (a tiny distance) of each other. This ensures the group effectively learns a single, coherent strategy.

Key Insights

  • Noise as a Feature, Not a Bug: Surprisingly, the "noise" introduced by gossiping (where agents have slightly different versions of the policy) actually seems to improve stability. It acts as a form of regularization, preventing the model from collapsing into poor local optima.
  • Zero Hyperparameter Tuning: One of the most impressive practical benefits is that GALA-A2C works using the default hyperparameters of vanilla A2C. Usually, distributed systems require grueling manual tuning to stay stable; GALA is robust enough to work out of the box.
  • Hardware Harmony: Traditional A2C often leaves GPUs idling while waiting for CPU-based simulators. GALA allows multiple learners to run in parallel on a single GPU, filling the "gaps" in computation and maximizing hardware throughput.

Key Results

The researchers evaluated GALA-A2C on the Atari-2600 benchmark, focusing on six diverse games (BeamRider, Breakout, Pong, Qbert, Seaquest, and SpaceInvaders) to test the algorithm's ability to handle different visual inputs and reward structures.

Comprehensive evaluation of GALA-A2C against baselines across sample complexity, computational complexity, and energy efficiency.
  • Stability at Scale: In a "simulator sweep," vanilla A2C's success rate plummeted as the number of simulators increased. GALA-A2C maintained or increased its success rate across all games.
  • Massive Speedups: GALA achieved a 10-fold improvement in GPU utilization over A2C. This translated to much higher frame rates (more data processed per second).
  • Energy Efficiency: GALA is significantly "greener." It processed roughly 20,000 more frames per Kilojoule than vanilla A2C, achieving higher game scores with less power draw.
  • Competitive Performance: GALA-A2C matched or outperformed state-of-the-art methods like IMPALA and A3C, often reaching higher scores in fewer training steps.

Limitations and Future Work

  • Topology Exploration: The paper primarily uses a directed ring graph for communication. While effective, different network shapes (like grids or fully connected graphs) might offer different trade-offs between speed and consensus.
  • On-Policy Focus: The current work focuses on on-policy methods (A2C). Extending gossip mechanisms to off-policy architectures (like DQN or Rainbow) could be a promising next step.
  • Large-Scale Clusters: While the paper shows great results on a single GPU with many simulators, testing GALA across massive multi-node clusters would further prove its scalability in industrial-scale AI training.

Summary updated

Abstract

Multi-simulator training has contributed to the recent success of Deep Reinforcement Learning (Deep RL) by stabilizing learning and allowing for higher training throughputs. In this work, we propose Gossip-based Actor-Learner Architectures (GALA) where several actor-learners (such as A2C agents) are organized in a peer-to-peer communication topology, and exchange information through asynchronous gossip in order to take advantage of a large number of distributed simulators. We prove that GALA agents remain within an epsilon-ball of one-another during training when using loosely coupled asynchronous communication. By reducing the amount of synchronization between agents, GALA is more computationally efficient and scalable compared to A2C, its fully-synchronous counterpart. GALA also outperforms A2C, being more robust and sample efficient. We show that we can run several loosely coupled GALA agents in parallel on a single GPU and achieve significantly higher hardware utilization and frame-rates than vanilla A2C at comparable power draws.
BibTeX
@inproceedings{assran2019gossip,
  author = {Mahmoud Assran and Joshua Romoff and Nicolas Ballas and Joelle Pineau and Mike Rabbat},
  title = {{Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=YzxbWkjPlE4},
}
Figure