Multi-agent Reinforcement Learning in OpenSpiel: A Reproduction Report

Michael Walton, Viliam LisýarXiv 2021 · arXiv

Abstract

This report provides a rigorous validation of the OpenSpiel framework by reproducing state-of-the-art (SOTA) results for several core multi-agent reinforcement learning (MARL) algorithms. By bridging the gap between OpenSpiel’s default configurations and the high-performance results reported in original research, the authors provide a comprehensive library of tuned hyperparameters, ensuring a reliable baseline for the research community.

Introduction

OpenSpiel is a powerful library for reinforcement learning in games, yet it often faces a "defaults" problem. Most out-of-the-box parameters are tuned for Kuhn Poker, a relatively simple environment. However, original research typically reports performance on more complex games like Leduc Poker. This discrepancy creates uncertainty: does an algorithm underperform due to implementation bugs or a lack of tuning? This study addresses this by validating OpenSpiel’s re-implementations against original SOTA results.

Methodology

The authors conducted an extensive reproduction study across fundamental MARL algorithms, including Fictitious Play (XFP, FSP), Counterfactual Regret Minimization (RCFR, DeepCFR), Policy-based methods (ED, NFSP), and meta-solvers (PSRO). To identify optimal configurations, they utilized Hyperband, a bandit-based hyperparameter optimization algorithm that adaptively allocates resources to find the best settings more efficiently than standard grid searches.

Key Insights and Reproductions

A significant finding involved Regression CFR (RCFR). While the original paper utilized regression trees, OpenSpiel employs neural networks. The study found that with a specific learning rate (0.001), the neural network version actually exceeded the performance of the original tree-based implementation.

Comparison of CFR and RCFR performance. The OpenSpiel neural network implementation of RCFR matches or exceeds original targets on Leduc Poker.

Conversely, Exploitability Descent (ED) proved difficult to scale. While it performed well on Kuhn Poker, the authors could not reproduce neural network results for Leduc Poker. They hypothesize that ED requires an "inner loop" modification—performing multiple gradient steps per iteration without re-computing the best response—to achieve reliable convergence in complex domains.

Exploitability Descent (ED) performance on Kuhn and Leduc. The neural network implementation (Neural Net ED) fails to converge on Leduc Poker compared to the tabular baseline.

Other algorithms showed strong alignment with SOTA:

  • DeepCFR: Once tuned with 15k–100k external sampling iterations, the implementation slightly improved over the original baseline.
  • NFSP: By matching the network architecture and exploration decay from the ARMAC paper, results closely tracked SOTA performance.
  • Policy Gradients: Algorithms like RMPG showed improved relative performance, suggesting a robust implementation in OpenSpiel.

Conclusion

This reproduction effort confirms that OpenSpiel is a high-quality framework for MARL research, provided that hyperparameters are appropriately tuned for the target environment. The authors have open-sourced their configurations and Docker images, providing the community with a verified set of baselines to accelerate future research in imperfect information games.

Summary updated

Abstract

In this report, we present results reproductions for several core algorithms implemented in the OpenSpiel framework for learning in games. The primary contribution of this work is a validation of OpenSpiel's re-implemented search and Reinforcement Learning algorithms against the results reported in their respective originating works. Additionally, we provide complete documentation of hyperparameters and source code required to reproduce these experiments easily and exactly.
BibTeX
@misc{walton2021multi,
  author = {Michael Walton and Viliam Lisý},
  title = {{Multi-agent Reinforcement Learning in OpenSpiel: A Reproduction Report}},
  year = {2021},
  eprint = {2103.00187},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.00187},
}
Figure