An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks

George Papadopoulos, Andreas Kontogiannis, Foteini Papadopoulou, Chaido Poulianou, Ioannis Koumentis, George Vouros

TL;DR

This paper exposes a critical blind spot in Multi-Agent Reinforcement Learning (MARL) evaluation: algorithms that achieve state-of-the-art (SoTA) status on popular team-sports or combat benchmarks often fail dramatically in truly cooperative, real-world scenarios. To fix this, the authors introduce PyMARLzoo+, an open-source framework that expands benchmarking across eight diverse, fully cooperative environments—including high-dimensional, image-based tasks—revealing that standard baselines frequently outperform complex, specialized algorithms.

What Problem Does This Paper Address?

Currently, the MARL community heavily relies on benchmarks like the StarCraft Multi-Agent Challenge (SMAC) and Google Research Football (GRF) to evaluate algorithms. However, the authors argue that these environments are cooperative-competitive (a team of learning agents playing against fixed AI bots) and don't adequately measure the subtle nuances of fully cooperative tasks.

Real-world applications—like multi-robot warehouse logistics, search and rescue, and factory automation—require intense joint exploration, precise action timing, task allocation, and handling sparse rewards without an opponent to react against. Furthermore, most MARL evaluations operate on low-dimensional, tabular state spaces, leaving algorithm performance on high-dimensional visual inputs (like camera images) largely unstudied.

Methods

The authors conduct a massive empirical study evaluating 11 prominent MARL algorithms under the Centralized Training with Decentralized Execution (CTDE) paradigm. The algorithms span value-based, actor-critic, transformer-based, and intrinsic exploration-focused methods:

  • Baselines & Value Decomposition: QMIX, QPLEX, MAA2C, MAPPO, HAPPO, COMA, MAT-DEC
  • Exploration-driven SoTA: EMC, MASER, EOI, CDS

To address visual observations, they integrate and compare two approaches for image-based tasks: training standard Convolutional Neural Networks (CNNs) from scratch versus utilizing frozen, pre-trained vision models (ResNet18, CLIP, and SlimSAM) as static feature encoders.

To make this extensive evaluation possible and reproducible, they developed and open-sourced PyMARLzoo+, extending the popular (E)PyMARL libraries to seamlessly support a wide array of new cooperative environments.

Datasets

The benchmark covers eight distinct fully cooperative environments, scaling across various difficulties, agent counts, and observation types:

  1. PettingZoo (Image-based): Entombed Cooperative, Pistonball, Cooperative Pong.
  2. Overcooked: Cramped Room, Asymmetric Advantages, Coordination Ring.
  3. PressurePlate: Sparse-reward grid worlds requiring sequential room unlocking.
  4. Multi-agent Particle Environment (MPE): The classic "Spread" navigation task.
  5. Level Based Foraging (LBF): Coordinated item collection based on agent levels.
  6. Multi-Robot Warehouse (RWARE): Grid-world robotics simulation evaluated in "hard" mode.
  7. Capture Target & Box Pushing: Tasks requiring simultaneous, synchronized agent actions.

Key Insights and Ideas

  • The SoTA Illusion: Algorithms designed to maximize exploration in SMAC (like MASER and EMC) underperform drastically in sparse-reward, fully cooperative tasks. Why? They rely heavily on Q-values to generate intrinsic rewards for sub-goals. In purely cooperative settings with highly sparse rewards, agents rarely see positive reinforcement, causing the algorithms to chase misleading, Q-value-driven mirages.
  • Value Decomposition Bottlenecks: Value decomposition methods (QMIX, QPLEX) tend to converge to suboptimal policies as the number of agents scales up, falling behind actor-critic methods like MAPPO and MAA2C in complex environments like Spread and PressurePlate.
  • The Power of Frozen Vision: For image-based observations, using a frozen, pre-trained ResNet18 encoder yields significantly more stable convergence and higher policy performance than training a CNN from scratch, while drastically slashing training time and memory overhead.

Key Results

The aggregate performance across the benchmarks highlights a surprising trend: standard, well-tuned baselines are incredibly robust.

Figure 1: Aggregated Normalized episodic rewards of each benchmark.

MAPPO, MAA2C, and QPLEX demonstrated the most consistent performance overall. In the challenging RWARE tasks, MAPPO was outperformed by HAPPO, MAT-DEC, and CDS. Conversely, COMA consistently ranked among the worst-performing algorithms across almost all environments. The paper also uniquely documents wall-clock training times, showing that off-policy methods take substantially longer to train due to large replay buffer processing overhead.

Limitations and Future Work

The benchmark successfully identified several "brick walls" for current MARL technology. Two environments in particular—Entombed Cooperative (PettingZoo) and the Coordination Ring layout (Overcooked)—proved entirely impossible to solve, with all 11 algorithms failing to find any effective policy.

Additionally, scaling up agent counts (beyond 4 agents) in sparse-reward environments without explicit communication channels remains a major open challenge for the field. The authors point to these unsolved tasks as the ideal proving grounds for the next generation of cooperative MARL algorithms.

Summary updated

Abstract

Multi-Agent Reinforcement Learning (MARL) has recently emerged as a significant area of research. However, MARL evaluation often lacks systematic diversity, hindering a comprehensive understanding of algorithms' capabilities. In particular, cooperative MARL algorithms are predominantly evaluated on benchmarks such as SMAC and GRF, which primarily feature team game scenarios without assessing adequately various aspects of agents' capabilities required in fully cooperative real-world tasks such as multi-robot cooperation and warehouse, resource management, search and rescue, and human-AI cooperation. Moreover, MARL algorithms are mainly evaluated on low dimensional state spaces, and thus their performance on high-dimensional (e.g., image) observations is not well-studied. To fill this gap, this paper highlights the crucial need for expanding systematic evaluation across a wider array of existing benchmarks. To this end, we conduct extensive evaluation and comparisons of well-known MARL algorithms on complex fully cooperative benchmarks, including tasks with images as agents' observations. Interestingly, our analysis shows that many algorithms, hailed as state-of-the-art on SMAC and GRF, may underperform standard MARL baselines on fully cooperative benchmarks. Finally, towards more systematic and better evaluation of cooperative MARL algorithms, we have open-sourced PyMARLzoo+, an extension of the widely used (E)PyMARL libraries, which addresses an open challenge from [TBG++21], facilitating seamless integration and support with all benchmarks of PettingZoo, as well as Overcooked, PressurePlate, Capture Target and Box Pushing.
BibTeX
@inproceedings{papadopoulos2025extended,
  author = {George Papadopoulos and Andreas Kontogiannis and Foteini Papadopoulou and Chaido Poulianou and Ioannis Koumentis and George Vouros},
  title = {{An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks}},
  booktitle = {AAMAS},
  year = {2025},
  doi = {10.48550/arxiv.2502.04773},
}
Figure