Distributed Deep Reinforcement Learning: An Overview

Mohammad Reza Samsami, Hossein AlimadadarXiv 2020 · arXiv

Abstract

Distributed Deep Reinforcement Learning (DRL) architectures have emerged as a powerful solution to the "experience bottleneck" inherent in traditional sequential RL. By decoupling data collection from the learning process and leveraging massive parallelism, these frameworks allow agents to process years of simulated experience in mere hours. This paper surveys the evolution of distributed DRL, from early parameter-server models to modern centralized-inference systems, and discusses the key innovations—such as V-trace and prioritized replay—that have led to superhuman performance across diverse benchmarks.

Introduction: The Experience Bottleneck

Deep Reinforcement Learning has achieved remarkable success in complex domains like Go and StarCraft II. However, standard RL faces a fundamental hurdle: the experience bottleneck. In a sequential setup, an agent interacts with an environment, collects data, and then updates its policy. This process is slow because the agent spends the majority of its time waiting for environment simulations. Furthermore, a single agent is prone to getting stuck in local optima due to limited exploration. To tackle complex real-world problems, distributed architectures are necessary to generate high-volume, diverse data and accelerate the learning cycle.

Distributed Architectures and Methods

The evolution of distributed DRL is marked by several landmark architectures that progressively improved scalability and efficiency.

GORILA and Ape-X

One of the earliest frameworks, GORILA (General Reinforcement Learning Architecture), introduced the use of a Parameter Server. It splits the workload into multiple actors that interact with environments and learners that calculate gradient updates, with a central server maintaining the master neural network. Later, Ape-X refined this by focusing on "Prioritized Experience Replay" in a distributed setting, ensuring that the learner prioritizes the most informative transitions collected by the actor fleet.

A distributed DRL architecture utilizing a parameter server to synchronize networks across actors and learners.

A3C and A2C

The Asynchronous Advantage Actor-Critic (A3C) moved away from massive clusters to utilize multi-core CPUs on a single machine. Multiple agents run in parallel, each with its own environment copy, updating a global network asynchronously. Its synchronous counterpart, A2C, waits for all agents to complete their steps before updating, which often results in more stable learning trajectories.

IMPALA: Decoupling Acting from Learning

The Importance Weighted Actor-Learner Architecture (IMPALA) introduced a major shift by completely decoupling "acting" from "learning." In this model, actors simply stream trajectories of experience to a central learner.

The decoupled actor-learner architecture where actors stream experience trajectories to a central learner.

Because actors might be using a slightly outdated policy compared to the learner's current state, IMPALA utilizes V-trace, a novel off-policy correction method, to account for "policy lag."

SEED RL

The most modern approach, SEED RL, addresses hardware inefficiencies by moving neural network inference from the actors to the learner. By centralizing inference on powerful GPUs or TPUs, SEED RL reduces the computational burden on actor nodes (typically CPUs) and significantly minimizes communication lag, allowing for massive scaling.

Key Insights

  • Decoupling is Essential: Successful architectures separate the "doing" (acting) from the "thinking" (learning). This allows the number of actors to scale into the thousands without bottlenecking the learning process.
  • Addressing Policy Lag: In distributed systems, communication delays are inevitable. Mathematical corrections like V-trace are critical to align data collected by older policies with the current learner state.
  • Hardware Efficiency: Centralizing inference (as in SEED RL) allows for the use of specialized accelerators for the most intensive computations, while running environments on cheaper, general-purpose hardware.
  • Diversity through Parallelism: Scaling is not just about speed; it facilitates broader exploration. Different actors can employ varied exploration strategies, providing the learner with a richer dataset than any single agent could produce.

Results and Performance

Distributed DRL has set new standards for speed and performance:

  • Training Speed: A3C can outperform standard DQN agents in both training time and final score by leveraging just 16 CPU cores.
  • Throughput: SEED RL can achieve up to one million queries per second, enabling training at millions of frames per second.
  • Benchmarks: These advancements culminated in Agent57, the first DRL agent to outperform human baselines on all 57 Atari games, achieved through a combination of distributed architecture and sophisticated exploration controllers.

Limitations and Future Directions

Despite their success, distributed systems face challenges. Communication overhead remains a primary bottleneck as systems scale to thousands of nodes. Additionally, the implementation complexity and high operational costs of these systems can be prohibitive. Future research is focused on making these distributed setups more accessible and improving their efficiency for multi-task learning environments.

Summary updated

BibTeX
@misc{samsami2020distributed,
  author = {Mohammad Reza Samsami and Hossein Alimadad},
  title = {{Distributed Deep Reinforcement Learning: An Overview}},
  year = {2020},
  eprint = {2011.11012},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.11012},
}
Figure