Abstract
This report provides a rigorous validation of the OpenSpiel framework by reproducing state-of-the-art (SOTA) results for several core multi-agent reinforcement learning (MARL) algorithms. By bridging the gap between OpenSpiel’s default configurations and the high-performance results reported in original research, the authors provide a comprehensive library of tuned hyperparameters, ensuring a reliable baseline for the research community.
Introduction
OpenSpiel is a powerful library for reinforcement learning in games, yet it often faces a "defaults" problem. Most out-of-the-box parameters are tuned for Kuhn Poker, a relatively simple environment. However, original research typically reports performance on more complex games like Leduc Poker. This discrepancy creates uncertainty: does an algorithm underperform due to implementation bugs or a lack of tuning? This study addresses this by validating OpenSpiel’s re-implementations against original SOTA results.
Methodology
The authors conducted an extensive reproduction study across fundamental MARL algorithms, including Fictitious Play (XFP, FSP), Counterfactual Regret Minimization (RCFR, DeepCFR), Policy-based methods (ED, NFSP), and meta-solvers (PSRO). To identify optimal configurations, they utilized Hyperband, a bandit-based hyperparameter optimization algorithm that adaptively allocates resources to find the best settings more efficiently than standard grid searches.
Key Insights and Reproductions
A significant finding involved Regression CFR (RCFR). While the original paper utilized regression trees, OpenSpiel employs neural networks. The study found that with a specific learning rate (0.001), the neural network version actually exceeded the performance of the original tree-based implementation.
Conversely, Exploitability Descent (ED) proved difficult to scale. While it performed well on Kuhn Poker, the authors could not reproduce neural network results for Leduc Poker. They hypothesize that ED requires an "inner loop" modification—performing multiple gradient steps per iteration without re-computing the best response—to achieve reliable convergence in complex domains.
Other algorithms showed strong alignment with SOTA:
- DeepCFR: Once tuned with 15k–100k external sampling iterations, the implementation slightly improved over the original baseline.
- NFSP: By matching the network architecture and exploration decay from the ARMAC paper, results closely tracked SOTA performance.
- Policy Gradients: Algorithms like RMPG showed improved relative performance, suggesting a robust implementation in OpenSpiel.
Conclusion
This reproduction effort confirms that OpenSpiel is a high-quality framework for MARL research, provided that hyperparameters are appropriately tuned for the target environment. The authors have open-sourced their configurations and Docker images, providing the community with a verified set of baselines to accelerate future research in imperfect information games.
Summary updated