Abstract
In complex, intransitive games like poker, no single strategy is universally dominant. This study investigates methods for combining "expert imitators"—AI agents trained to replicate the styles of top-tier bots—to enhance performance. We evaluate ensemble voting and dynamic strategy selection using multi-armed bandits (UCB1) in both Limit and No-Limit Texas Hold'em. Our results indicate that dynamic selection significantly improves performance in Limit poker by adapting to opponent styles, whereas the high variance of No-Limit poker poses substantial challenges for selection algorithms.
Introduction
Poker is characterized by an "intransitive" nature, where strategies often follow a rock-paper-scissors dynamic. Consequently, a static AI agent may struggle against diverse opponents. This research explores whether a library of expert imitators, developed using Case-Based Reasoning (CBR) via the Sartre system, can be effectively combined to create a more robust player. We contrast two approaches: a static Ensemble method and a Dynamic Selection method that treats the choice of expert as a Multi-Armed Bandit problem.
Methodology
The expert imitators were trained on hand histories from the 2010 AAAI Computer Poker Competition. To manage the inherent noise in poker outcomes, we utilized DIVAT (Ignorant Value Assessment Tool) for variance reduction. DIVAT estimates the expected value (EV) of decisions relative to a baseline, providing a cleaner signal for the selection algorithm than raw profit/loss.
The dynamic selector employed the UCB1 algorithm to balance exploration and exploitation. In each hand, the selector chooses an imitator based on its past performance against the current opponent.
Key Results
Limit Texas Hold'em
In the Limit domain, the Dynamic selection approach emerged as the superior strategy. It achieved an average profit of 0.35095 small bets per hand, outperforming both individual experts and the ensemble approach. The selection process demonstrates how the UCB1 algorithm converges on specific experts over time when facing a Nash-equilibrium bot like Fell Omen 2.
No-Limit Texas Hold'em
The transition to No-Limit poker introduced significant variance. While the Dynamic and Ensemble approaches remained competitive, they were slightly outperformed by the strongest individual expert, Hyperborean-max. The "all-in" mechanics and frequent folding in No-Limit reduce the availability of showdown data, which DIVAT relies on for effective variance reduction. The selection behavior against SimpleBot shows the algorithm attempting to identify the best expert amidst noisier feedback.
Discussion and Conclusion
The study demonstrates that dynamic strategy selection is highly effective in environments where variance can be controlled, such as Limit poker. However, the "Variance Trap" in No-Limit poker remains a hurdle. The UCB1 algorithm occasionally favored "safer," lower-variance imitators over more profitable but volatile ones. Future work should focus on improving variance reduction techniques that do not depend exclusively on showdown information and testing against a broader spectrum of opponent types to further validate the benefits of dynamic adaptation.
Summary updated