ScrofaZero: Mastering Trick-taking Poker Game Gongzhu by Deep Reinforcement Learning

Naichen Shi, Ruichen Li, Sun YouranarXiv 2021 · OpenReview

Abstract: While AI has made remarkable progress in perfect-information games, trick-taking poker remains a significant challenge due to the high level of reasoning and inference required. We present ScrofaZero, a deep reinforcement learning agent trained from tabula rasa to master Gongzhu, a complex Chinese trick-taking game. By introducing novel techniques such as stratified sampling and Integral over Equivalent Class (IEC) for Bayesian inference, ScrofaZero achieves human expert-level performance. Our methodology provides a transferable framework for solving a wide range of imperfect-information trick-taking games.

The Challenge of Trick-Taking Games

Artificial Intelligence has conquered perfect-information domains like Chess and Go, but imperfect-information games—where players have hidden private information—remain a frontier. Trick-taking games like Hearts, Spades, and Contract Bridge are particularly difficult because they require long-term strategic planning coupled with continuous logical deduction about opponents' hidden hands.

This research utilizes Gongzhu as a benchmark. Gongzhu is more complex than toy poker variants but computationally more manageable than massive real-time strategy games. The core difficulty lies in sequential decision-making under high noise: an agent must infer what cards opponents hold based on play history and use that knowledge to navigate a massive hidden information space.

Methodology: Learning from Scratch

ScrofaZero is trained entirely via self-play without human data, utilizing an architecture inspired by AlphaZero that combines neural networks with Monte Carlo Tree Search (MCTS). To handle the complexities of hidden information, the authors implemented several key innovations:

  • Perfect Information Training ("Double Dummy"): During training, the network is allowed to see all cards. This "cheating" eliminates noise, allowing the network to learn deep strategic representations much faster.
  • Stratified Sampling: The number of possible hidden card configurations is astronomical (up to $10^{46}$). To manage this, the agent identifies "key cards"—those with high-variance that significantly impact game outcomes—and exhaustively samples their configurations while randomly sampling the rest.
Variance of card values across suits, showing the high strategic importance of 'key cards' like the Queen of Spades (SQ).
  • Integral over Equivalent Class (IEC): Standard Bayesian inference is computationally intractable in this domain. IEC provides a mathematical shortcut by identifying "irrelevant cards" (low-value cards whose exact placement doesn't change the game's math) and grouping them into equivalent classes. This allows the AI to accurately weight the probability of different scenarios without excessive computation.

Evaluation and Results

ScrofaZero was evaluated against a suite of heuristic bots, including "Mr. Greed," a highly tuned expert-level bot using partial backward induction.

Training progress of the ScrofaZero raw network against baseline bots, eventually surpassing the expert-level Mr. Greed.

Key findings include:

  • Expert-Level Performance: ScrofaZero decisively outperformed Mr. Greed, achieving a 67.7% win/draw rate.
  • The Power of IEC: While Stratified Sampling (SS) alone yielded a +59.7 Winning Point over Mr. Greed (WPG), adding the novel IEC technique boosted performance significantly to +73.9 WPG.
  • Emergent Heuristics: Without hardcoded rules, the neural network independently learned to assign risk/reward values to specific cards that mirror human expert strategies, such as identifying the Queen of Spades as a primary hazard.
  • Compute Efficiency: Remarkably, the entire training process was completed on just two Nvidia 2080Ti GPUs, demonstrating that sophisticated game AI does not always require massive server farms.

Limitations and Future Work

Training under perfect information, while efficient, may limit the AI's ability to learn advanced psychological tactics like bluffing, as the training environment assumes full transparency. Future research will focus on incorporating the sampling process directly into the neural network architecture and applying this framework to more complex challenges like Contract Bridge. These techniques for distilling information from history have broad potential applications in real-world multi-agent decision-making.

Summary updated

Abstract

People have made remarkable progress in game AIs, especially in domain of perfect information game. However, trick-taking poker game, as a popular form of imperfect information game, has been regarded as a challenge for a long time. Since trick-taking game requires high level of not only reasoning, but also inference to excel, it can be a new milestone for imperfect information game AI. We study Gongzhu, a trick-taking game analogous to, but slightly simpler than contract bridge. Nonetheless, the strategies of Gongzhu are complex enough for both human and computer players. We train a strong Gongzhu AI ScrofaZero from \textit{tabula rasa} by deep reinforcement learning, while few previous efforts on solving trick-taking poker game utilize the representation power of neural networks. Also, we introduce new techniques for imperfect information game including stratified sampling, importance weighting, integral over equivalent class, Bayesian inference, etc. Our AI can achieve human expert level performance. The methodologies in building our program can be easily transferred into a wide range of trick-taking games.
BibTeX
@misc{shi2021scrofazero,
  author = {Naichen Shi and Ruichen Li and Sun Youran},
  title = {{ScrofaZero: Mastering Trick-taking Poker Game Gongzhu by Deep Reinforcement Learning}},
  year = {2021},
  eprint = {2102.07495},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.07495},
}
Figure