JaxMARL: Multi-Agent RL Environments in JAX

Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Garðar Ingvarsson, Timon Willi, Akbir Khan, Christian Schröder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whiteson, Bruno Lacerda, Nick Hawes, Tim Rocktäschel, Chris Lu, Jakob Nicolaus Foerster

Introduction

Multi-Agent Reinforcement Learning (MARL) is notoriously computationally expensive. Unlike single-agent RL, MARL agents must navigate non-stationarity and decentralized partial observability. Traditionally, RL environments run on the CPU while neural networks train on the GPU, creating a massive communication bottleneck. JaxMARL is the first open-source library to bring the massive parallelization of JAX to MARL, running both environments and training pipelines entirely on the GPU. This approach achieves speedups of up to 12,500x, enabling researchers to run thousands of experiments in the time it previously took to run one.

Addressing the Evaluation Crisis

Because experiments are traditionally slow, the MARL field suffers from an "evaluation crisis." A survey cited in the paper found that 50% of recent MARL papers only tested on a single environment, leading to "environment overfitting." JaxMARL aims to solve this by making large-scale, multi-environment evaluation computationally feasible for researchers with modest academic hardware.

Methods: The Power of JAX

The library leverages two core JAX features to eliminate the "CPU-GPU tax":

  1. jit (Just-In-Time compilation): Compiles Python/JAX code into highly optimized XLA kernels for the GPU.
  2. vmap (Vectorized Mapping): Allows the library to vectorize the entire training pipeline. Instead of running one environment, JaxMARL can run thousands of environment instances—and even thousands of independent training runs with different seeds—in parallel on a single GPU.

Datasets and Benchmarks

JaxMARL includes JAX-based implementations of several "gold standard" MARL benchmarks and introduces two new ones designed for high-performance research.

Overview of the diverse MARL environments supported by JaxMARL, including SMAX, STORM, and reimplementations of MPE and Hanabi.
  • SMAX (New): A JAX-native reimplementation of the StarCraft Multi-Agent Challenge (SMAC). It mimics StarCraft II dynamics without the heavy game engine, making it orders of magnitude faster.
  • STORM (New): Inspired by DeepMind’s Melting Pot, this environment focuses on "social dilemmas" in grid-world scenarios.
  • Existing Reimplementations: The library includes JAX versions of MPE (Particle environments), Hanabi (card game), Overcooked (cooperative cooking), and MABrax (continuous multi-agent robotics).

Key Results and Insights

The performance gains are transformative for the research cycle:

  • Vectorized Speedup: When running multiple training runs in parallel, JaxMARL is up to 12,500x faster than traditional CPU-based pipelines.
  • SMAX vs. SMAC: SMAX training can be up to 40,000x faster than the original SMAC implementation when multiple runs are vectorized on a single NVIDIA 2080 GPU.
  • MPE Benchmarking: Training a Q-learning agent on MPE's "Simple Spread" task takes just 130 seconds in JaxMARL, compared to over an hour in the popular PyMARL library.

By lowering the cost of experiments, JaxMARL removes the excuse for reporting results on only one or two maps, encouraging more robust and statistically significant MARL research.

Limitations and Future Work

While JaxMARL is a massive leap forward, some hurdles remain. Speedups are currently more pronounced for on-policy methods (like PPO) than off-policy methods (like Q-learning), as the latter require large replay buffers that consume significant GPU memory. Additionally, JAX's requirement for static array shapes makes it difficult to handle environments where the number of agents changes dynamically. Future work will focus on developing entirely new MARL challenges that specifically push the boundaries of hardware-accelerated libraries.

Summary updated

BibTeX
@inproceedings{rutherford2024jaxmarl,
  author = {Alexander Rutherford and Benjamin Ellis and Matteo Gallici and Jonathan Cook and Andrei Lupu and Garðar Ingvarsson and Timon Willi and Akbir Khan and Christian Schröder de Witt and Alexandra Souly and Saptarashmi Bandyopadhyay and Mikayel Samvelyan and Minqi Jiang and Robert Tjarko Lange and Shimon Whiteson and Bruno Lacerda and Nick Hawes and Tim Rocktäschel and Chris Lu and Jakob Nicolaus Foerster},
  title = {{JaxMARL: Multi-Agent RL Environments in JAX}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=F1tCj0t9XJ},
}
Figure