JaxMARL: Multi-Agent RL Environments in JAX

Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Garðar Ingvarsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whiteson, Bruno Lacerda, Nick Hawes, Tim Rocktäschel, Chris Lu, Jakob Nicolaus Foerster

TL;DR

JaxMARL is the first open-source library that unifies a broad suite of Multi-Agent Reinforcement Learning (MARL) environments and baseline algorithms entirely in JAX. By running both the simulation and policy updates directly on hardware accelerators (GPUs/TPUs), it achieves simulation speedups of up to 8,500× and cuts training times from days to minutes.

What Problem Does This Paper Address?

Multi-Agent Reinforcement Learning is notoriously compute-intensive. Coordinating multiple learning agents introduces non-stationarity and partial observability, drastically worsening sample complexity. Traditionally, researchers run environment simulations on the CPU and push experience batches to the GPU for network updates. This architecture causes a severe data-transfer bottleneck over PCIe buses and limits parallelization unless researchers have access to massive CPU clusters.

Because running experiments is slow and expensive, the MARL field suffers from an "evaluation crisis": roughly 80% of published papers benchmark on only one or two environments (often selectively reported SMAC or MPE maps), causing algorithmic overfitting. JaxMARL directly addresses these computational barriers and fragmented toolkits by providing an end-to-end, hardware-accelerated MARL pipeline in pure Python.

Methods

  • End-to-End Accelerator Pipeline: Built using JAX, JaxMARL leverages JIT (just-in-time) compilation via XLA and vmap (vectorization). Environment steps and neural network gradient updates happen in-device on the GPU/TPU, eliminating host-to-device communication bottlenecks.
  • Unified PettingZoo-Style Functional API: To accommodate JAX's functional programming paradigm, environment methods (reset, step) take explicit PRNG keys and immutable state objects. Because XLA requires static array dimensions, environments handle variable or terminating agents using static maximum shapes and dummy action masking rather than dynamic agent iterators.
  • SMAX (GPU-Native StarCraft Benchmark): The authors created SMAX, a vectorized 2D continuous-space reimplementation of the StarCraft Multi-Agent Challenge (SMAC/SMACv2). It strips away the slow, CPU-bound StarCraft II engine while preserving sight/attack ranges, weapon cooldowns, unit interactions, and procedural scenario generation. It also implements a fully decentralized heuristic AI and supports symmetric competitive self-play.
  • Single-File Baseline Implementations: Following the CleanRL philosophy, JaxMARL provides transparent, single-file implementations of leading algorithms: Independent PPO (IPPO, feed-forward and RNN), Value Decomposition Networks (VDN), QMIX, and Independent Q-Learning (IQL).

Datasets

Rather than static datasets, the library provides eight standard MARL simulation suites:

  • Centralised Training with Decentralised Execution (CTDE): Multi-Agent Particle Environments (MPE) and Multi-Agent Brax (MABrax continuous control robotics based on MuJoCo).
  • Zero-Shot Coordination (ZSC): Overcooked and Hanabi (implemented natively in Python rather than C++).
  • General-Sum Dilemmas: Coin Game and STORM (Spatial-Temporal Representations of Matrix Games, inspired by DeepMind's Melting Pot).
  • Communication & Unit Benchmarking: Switch Riddle (debugging) and SMAX (incorporating classic scenarios like 2s3z, 27m_vs_30m, and SMACv2 procedural setups).

Key Insights and Ideas

  • Vectorization Overcomes Single-Threaded Latency: While executing a single environment step in JAX can be slower than low-level C++ CPU code, running thousands of vectorized environments in parallel on a single GPU scales almost linearly, yielding tens of millions of steps per second.
  • Liberating SMAC from StarCraft II: SMAC traditionally requires running the full commercial StarCraft II binary, making GPU parallelization impossible and locking scenarios to rigid engine rules (e.g., no cross-race teams). SMAX demonstrates that geometric 2D approximations retain the strategic depth of unit micromanagement while running entirely on-chip.
  • Alleviating the Evaluation Crisis: Lowering experiment runtimes from 40+ hours down to 30 minutes on consumer/standard academic hardware allows researchers to reliably sweep hyperparameters, evaluate across many random seeds, and benchmark across diverse environments instead of cherry-picking a single domain.

Key Results

  • Environment Simulation Throughput: When running 10,000 environments in parallel on a single NVIDIA A100 GPU:
  • Overcooked reaches 16.9 million steps/sec (>8,500× faster than the single-threaded CPU version).
  • MPE Simple Spread reaches 39.9 million steps/sec.
  • SMAX (2s3z) reaches 2.71 million steps/sec (compared to 83 steps/sec for SMAC on CPU).
  • End-to-End Training Speed:
  • Vectorized IPPO on MPE trains 1,024 agents in 198.4 seconds (<0.2 seconds per agent), an over 12,500× throughput speedup compared to MARLLIB on identical hardware.
  • Training 512 SMAX agents on 2s3z completes in under 33 minutes, compared to ~44 hours for a single PyTorch SMAC run.
  • Validation & Correctness: Policies trained in JaxMARL transferred zero-shot to original PettingZoo environments with identical trajectory rollouts, and value-based baselines (QMIX, VDN, IQL) matched or outperformed reported PyMARL returns.

Limitations and Future Work

  • Simplified Dynamics in SMAX: To keep SMAX mathematically lean for XLA compilation, certain StarCraft mechanics were omitted—Zerg passive health regeneration and Protoss recharging shields were removed (compensated with raw health pools), and complex units (Medivacs, Banelings, Colossi) are currently excluded.
  • Static Shape Constraints: JAX's compilation model forbids dynamically changing array shapes, requiring fixed limits on agent counts per episode and dummy action padding.
  • Scope of Supported Algorithms: While core value-based and policy gradient algorithms are provided, several modern actor-critic variants (e.g., MAPPO with continuous action spaces, off-policy actor-critic) and advanced population-based training methods remain natural directions for community expansion.

Summary updated

Abstract

Benchmarks play an important role in the development of machine learning algorithms. Reinforcement learning environments are traditionally run on the CPU, limiting their scalability with typical academic compute. However, recent advancements in JAX have enabled the wider use of hardware acceleration to overcome these computational hurdles by producing massively parallel RL training pipelines and environments. This is particularly useful for multi-agent reinforcement learning (MARL) research where not only multiple agents must be considered at each environment step, adding additional computational burden, but also the sample complexity is increased due to non-stationarity, decentralised partial observability, or other MARL challenges. In this paper, we present JaxMARL, the first open-source code base that combines ease-of-use with GPU enabled efficiency, and supports a large number of commonly used MARL environments as well as popular baseline algorithms. Our experiments show that our JAX-based implementations are up to 1400x faster than existing single-threaded baselines. This enables efficient and thorough evaluations, with the potential to alleviate the *evaluation crisis* of the field. We also introduce and benchmark SMAX, a vectorised, simplified version of the StarCraft Multi-Agent Challenge, which removes the need to run the StarCraft II game engine. This not only enables GPU acceleration, but also provides a more flexible MARL environment, unlocking the potential for self-play, meta-learning, and other future applications in MARL.
BibTeX
@inproceedings{rutherford2024jaxmarl,
  author = {Alexander Rutherford and Benjamin Ellis and Matteo Gallici and Jonathan Cook and Andrei Lupu and Garðar Ingvarsson and Timon Willi and Akbir Khan and Christian Schroeder de Witt and Alexandra Souly and Saptarashmi Bandyopadhyay and Mikayel Samvelyan and Minqi Jiang and Robert Tjarko Lange and Shimon Whiteson and Bruno Lacerda and Nick Hawes and Tim Rocktäschel and Chris Lu and Jakob Nicolaus Foerster},
  title = {{JaxMARL: Multi-Agent RL Environments in JAX}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=F1tCj0t9XJ},
}
Figure