Enhanced DouDiZhu Card Game Strategy Using Oracle Guiding and Adaptive Deep Monte Carlo Method

Qian Luo, Tien Ping Tan, Daochen Zha, Tianqiao ZhangIJCAI 2024

Introduction

DouDiZhu is a popular three-player adversarial card game that presents a significant challenge for game AI research. Unlike perfect information games like Chess or Go, DouDiZhu is an imperfect information game where players cannot see their opponents' cards. This creates three major hurdles for standard Deep Reinforcement Learning (DRL): extreme uncertainty in decision-making, a vast state-action space ($10^{83}$ states), and sparse rewards provided only at the end of a match. While previous models like DouZero utilized Deep Monte Carlo (DMC) methods, they often suffered from training instability and overfitting to high-variance outcomes.

Methodology

The proposed framework, OADMCDou, introduces two primary innovations to address these challenges: Oracle Guiding and Adaptive Deep Monte Carlo (ADMC).

Oracle Guiding

To overcome the "slow start" problem in reinforcement learning, the researchers developed an Oracle agent. Initially, this agent is granted perfect information, allowing it to see all cards in the game. This omniscient perspective enables the agent to quickly learn the underlying logic of optimal play. As training progresses, a dropout coefficient ($\lambda$) is used to gradually reduce the hidden information available to the agent. Eventually, the Oracle transitions into a standard agent that relies only on legally observable information, retaining the strategic intuition developed during the perfect-information phase.

The OADMCDou framework architecture, illustrating the interaction between the Actor, Learner, and the Oracle Guiding mechanism.

Adaptive Deep Monte Carlo (ADMC)

Standard DMC methods can be erratic in high-variance environments. A single win due to a lucky hand can lead to large gradient updates that destabilize a previously sound policy. ADMC introduces a gradient weight clipping mechanism, inspired by Proximal Policy Optimization (PPO). By constraining the ratio between new and old Q-values within a predefined range (controlled by hyperparameter $\gamma$), ADMC ensures that no single game outcome causes a destructive shift in strategy, promoting long-term stability.

Experimental Setup

The model was trained via self-play on a single server using 7 actor processes, generating over 100 million state-action pairs daily. Evaluation was conducted against established baselines, including DouZero and WagerWin, using a set of 10,000 randomly dealt decks. To eliminate the influence of luck, each match was played twice with roles swapped.

Results

OADMCDou demonstrated superior performance and stability compared to existing state-of-the-art models.

Comparison of performance (ADP) and training stability (ADP Loss) between ADMCDou and the baseline DouZero*.
  • Performance: OADMCDou achieved an Average Difference in Points (ADP) of 0.104 ± 0.041 over DouZero, a statistically significant improvement.
  • Stability: The framework achieved a 28.6% reduction in Mean Squared Error (MSE) loss compared to the baseline, validating the effectiveness of the ADMC clipping mechanism.
  • Win Rates: In head-to-head competitions, OADMCDou consistently outperformed other models, including "DouZero+," which incorporates opponent modeling.

Conclusion and Future Work

OADMCDou effectively addresses the instability and complexity of DouDiZhu through curriculum-based information sharing and adaptive gradient updates. Future research will focus on integrating the "bidding" phase of the game and extending the Oracle Guiding approach to other complex imperfect information games such as Mahjong and Contract Bridge. Additionally, the team plans to explore Transformer-based architectures to further enhance state representation.

Summary updated

BibTeX
@inproceedings{luo2024enhanced,
  author = {Qian Luo and Tien Ping Tan and Daochen Zha and Tianqiao Zhang},
  title = {{Enhanced DouDiZhu Card Game Strategy Using Oracle Guiding and Adaptive Deep Monte Carlo Method}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure