On Combining Decisions from Multiple Expert Imitators for Performance

Jonathan Rubin, Ian D. Watson

Abstract

In complex, intransitive games like poker, no single strategy is universally dominant. This study investigates methods for combining "expert imitators"—AI agents trained to replicate the styles of top-tier bots—to enhance performance. We evaluate ensemble voting and dynamic strategy selection using multi-armed bandits (UCB1) in both Limit and No-Limit Texas Hold'em. Our results indicate that dynamic selection significantly improves performance in Limit poker by adapting to opponent styles, whereas the high variance of No-Limit poker poses substantial challenges for selection algorithms.

Introduction

Poker is characterized by an "intransitive" nature, where strategies often follow a rock-paper-scissors dynamic. Consequently, a static AI agent may struggle against diverse opponents. This research explores whether a library of expert imitators, developed using Case-Based Reasoning (CBR) via the Sartre system, can be effectively combined to create a more robust player. We contrast two approaches: a static Ensemble method and a Dynamic Selection method that treats the choice of expert as a Multi-Armed Bandit problem.

Methodology

The expert imitators were trained on hand histories from the 2010 AAAI Computer Poker Competition. To manage the inherent noise in poker outcomes, we utilized DIVAT (Ignorant Value Assessment Tool) for variance reduction. DIVAT estimates the expected value (EV) of decisions relative to a baseline, providing a cleaner signal for the selection algorithm than raw profit/loss.

The dynamic selector employed the UCB1 algorithm to balance exploration and exploitation. In each hand, the selector chooses an imitator based on its past performance against the current opponent.

Key Results

Limit Texas Hold'em

In the Limit domain, the Dynamic selection approach emerged as the superior strategy. It achieved an average profit of 0.35095 small bets per hand, outperforming both individual experts and the ensemble approach. The selection process demonstrates how the UCB1 algorithm converges on specific experts over time when facing a Nash-equilibrium bot like Fell Omen 2.

Total Imitator Selections against Fell Omen 2 (Limit)

No-Limit Texas Hold'em

The transition to No-Limit poker introduced significant variance. While the Dynamic and Ensemble approaches remained competitive, they were slightly outperformed by the strongest individual expert, Hyperborean-max. The "all-in" mechanics and frequent folding in No-Limit reduce the availability of showdown data, which DIVAT relies on for effective variance reduction. The selection behavior against SimpleBot shows the algorithm attempting to identify the best expert amidst noisier feedback.

Total Imitator Selections against SimpleBot (No-Limit)

Discussion and Conclusion

The study demonstrates that dynamic strategy selection is highly effective in environments where variance can be controlled, such as Limit poker. However, the "Variance Trap" in No-Limit poker remains a hurdle. The UCB1 algorithm occasionally favored "safer," lower-variance imitators over more profitable but volatile ones. Future work should focus on improving variance reduction techniques that do not depend exclusively on showdown information and testing against a broader spectrum of opponent types to further validate the benefits of dynamic adaptation.

Summary updated

Abstract

One approach for artificially intelligent agents wishing to maximise some performance metric in a given domain is to learn from a collection of training data that consists of actions or decisions made by some expert, in an attempt to imitate that expert's style. We refer to this type of agent as an expert imitator. In this paper we investigate whether performance can be improved by combining decisions from multiple expert imitators. In particular, we investigate two existing approaches for combining decisions. The first approach combines decisions by employing ensemble voting between multiple expert imitators. The second approach dynamically selects the best imitator to use at runtime given the performance of the imitators in the current environment. We investigate these approaches in the domain of computer poker. In particular, we create expert imitators for limit and no limit Texas Hold'em and determine whether their performance can be improved by combining their decisions using the two approaches listed above.
BibTeX
@inproceedings{rubin2011combining,
  author = {Jonathan Rubin and Ian D. Watson},
  title = {{On Combining Decisions from Multiple Expert Imitators for Performance}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2011},
  doi = {10.5591/978-1-57735-516-8/ijcai11-067},
}
Figure