Abstract
Solving large-scale imperfect-information games (IIGs) typically requires manual abstraction or neural approximation of regret values. However, existing deep learning approaches often struggle with state-value estimation and slow initial convergence. This paper presents Deep Dueling Counterfactual Regret Minimization (D2CFR), which introduces a dueling neural network architecture to decouple state values from action advantages and a time-shifted Monte Carlo rectification module to stabilize early training. Evaluated on Leduc Hold’em and Heads-up No-limit Texas Hold’em, D2CFR demonstrates superior convergence speed and lower exploitability compared to state-of-the-art baselines.
Introduction: The Challenge of Imperfect Information
Solving imperfect-information games (IIGs)—scenarios like Poker where players lack full knowledge of the game state—is a cornerstone of artificial intelligence research. The standard approach, Counterfactual Regret Minimization (CFR), traditionally relies on game abstraction. As shown in Figure 1, abstraction simplifies the game tree to make it computationally tractable, but this process requires significant domain expertise and inevitably leads to information loss.
Recent "Deep CFR" methods have attempted to replace manual abstraction with neural networks to approximate regret values. However, these methods often treat all actions equally and struggle to accurately estimate the value of specific game states. Furthermore, neural networks are prone to high inaccuracy during the early stages of the iterative CFR process, leading to slow convergence and suboptimal strategies.
Proposed Method: Deep Dueling CFR (D2CFR)
The authors introduce D2CFR to address these limitations through two primary architectural innovations: the Dueling Neural Network (DNet) and a Time-Shifted Monte Carlo Rectification module.
Dueling Neural Network (DNet)
Inspired by dueling architectures in reinforcement learning, the D2CFR value network is split into two distinct streams. One stream explicitly estimates the state value (the average utility of an information set), while the other estimates the action-state value (the advantage of a specific action). This decoupling allows the model to learn a more robust representation of the game's value landscape. In games like Poker, where different betting sizes might have very similar strategic values, this separation prevents the model from having to "re-learn" the base value of a state for every possible action.
Time-Shifted Monte Carlo Rectification
To solve the "cold start" problem where neural networks provide poor estimates early in training, the authors designed a rectification module. This module uses Monte Carlo (MC) simulations to provide "ground truth" state values. A weighting mechanism ($\alpha$ and $\beta$) shifts over time: early in the process, the model relies heavily on MC simulations; as training progresses and the network becomes more accurate, it transitions to relying on its own internal predictions.
Hybrid Loss Function
Beyond architectural changes, D2CFR employs a hybrid loss function. Instead of relying solely on Mean Squared Error (MSE), it combines MSE with KL-divergence to better regularize the policy network's learning, ensuring smoother strategy updates across iterations.
Experimental Results
The researchers evaluated D2CFR on two standard poker benchmarks: Leduc Hold’em (to measure exploitability) and Heads-up No-limit Texas Hold’em (HNLH) (to test real-world complexity).
- Lower Exploitability: In Leduc Hold’em, D2CFR achieved lower exploitability than both DeepCFR and Single DeepCFR (SD-CFR), indicating a strategy significantly closer to the theoretical Nash Equilibrium.
- Dominant Win Rates: In head-to-head HNLH matches, D2CFR defeated DeepCFR by 75.52 mbb/g (milli-big blinds per game) and outperformed the Neural Fictitious Self-Play (NFSP) baseline by 119.44 mbb/g.
- Faster Convergence: Ablation studies confirmed that the DNet architecture reduced policy loss compared to standard networks, and the rectification module effectively mitigated approximation errors during the critical early stages of training.
Conclusion and Future Work
D2CFR represents a significant step forward in solving IIGs without manual abstraction. By decoupling state values and action regrets, the algorithm achieves higher efficiency and better performance in massive-scale games. Future work will explore more sophisticated network architectures and test the algorithm's general-purpose utility in complex domains beyond Poker.
Summary updated