Opponent-Stratified Constrained GRPO for Safe Exploitation in Poker

A game-playing agent that follows an equilibrium strategy cannot be beaten on average, but it also wins little from opponents who make predictable mistakes. An agent that best-responds to a model of its opponent wins more against that opponent, but it can be crushed by anyone who notices what it is doing. The game-theory community has well-understood ways to trade these off, for example "restricted responses" that maximize winnings against an opponent model while capping how much a worst-case adversary could win back. These methods require exact solvers over the whole game tree, so they only run on tiny games. Language-model agents trained by reinforcement learning, on the other hand, scale to large games but have no mechanism for this tradeoff. The best-known language-model poker player, trained to imitate a solver, was badly exploited in live play.

We propose to train a language-model poker agent with Group Relative Policy Optimization (GRPO, a critic-free reinforcement learning method that scores each sampled answer against the other answers sampled for the same input). Each group of rollouts from a given deal is split between two opponents: the opponent model we want to exploit, and a learned counter-exploiter trained to beat the current agent. Advantages are normalized separately inside each opponent's half of the group. A Lagrange multiplier on the agent's measured exploitability sets how much the counter-exploiter's half counts. The resulting training objective is a restricted Nash response written in the form GRPO can optimize. We first validate it in small poker variants, where exploitability of the language model's actual policy can be computed exactly, and then test it in no-limit hold'em, where it cannot.

Core Claim

The obvious combination of these two directions is to take Constrained GRPO and set its cost to "exploitability." That paper adds Lagrangian constraints to GRPO and shows that subtracting a cost from the reward before group normalization distorts the multiplier. Its fix is to standardize reward and cost separately and then combine them. The combination above is ill-posed as stated. Exploitability is a property of a whole policy, not of any single trajectory, so there is no per-rollout cost to standardize.

Our claim is that the restricted-response structure resolves this. The expected payoff of a rollout played against a best response to the current policy is, up to the known game value, an unbiased sample of that policy's exploitability. So we can make exploitability a per-rollout quantity by changing who the opponent is in part of each group. The Lagrangian becomes

$$ \max_\pi \; \mathbb{E}[u(\pi, M)] + \lambda\left(\mathbb{E}[u(\pi, B_\pi)] - (v^* - \epsilon)\right), $$

where $u$ is the agent's chip payoff, $M$ is the fixed opponent model, $B_\pi$ is a best response (or learned approximation) to the current policy $\pi$, $v^*$ is the game value, $\epsilon$ is the exploitability budget, and $\lambda \ge 0$ is updated by dual ascent on measured exploitability.

For each deal, $G/2$ rollouts are played against $M$ and $G/2$ against $B_\pi$. Each half is standardized within itself, and the $B_\pi$ half is scaled by $\lambda$. This differs from Constrained GRPO, whose reward and cost live on the same trajectory. Here the two terms come from different rollouts with very different payoff variances. Winnings against an exploitable $M$ are large and noisy, while losses against $B_\pi$ are systematic. Pooled normalization would let the $M$ half dominate, which is exactly the variance-mismatch pathology Constrained GRPO proves, reappearing across opponents.

Relation to Prior Work

Three papers motivate the proposal.

PokerBench fine-tuned Llama-3-8B on 560,000 solver-labeled spots. It reached 78.26% exact-match accuracy yet lost to GPT-4 at 22.20 bb/100 (big blinds per hundred hands) in heads-up play, because aggressive betting exploited its passive style. The authors call for reinforcement learning. We supply the RL objective, and we add the counter-exploitation control that plain RL against a fixed opponent would lack.

Agents That Certify Their Own Exploits (CS-RNR) solves restricted games in which the opponent is pinned to a conservative model with probability $p$. It audits each candidate policy with an exact full-tree best response. It reports zero certificate violations over 36,000 audited hands and 6.2× the steady-state gain of a binary gate. It also shows that naively mixing a best response with Nash play reaches exploitability 2.04 against a 0.15 budget. We keep its core structure: a coherent restricted game rather than a mixture of trajectories. We replace the tabular solve and audit with policy-gradient training and a learned attacker, which is what lets the method leave Leduc.

The Cost of Blind Confidence (SWAP) blends an exploiting policy and a robust policy with a bandit and relies on dense rewards. We avoid online switching entirely and fold robustness into training.

Constrained GRPO supplies the separate-standardization technique. Restricted Nash Response supplies the objective. The GRPO stability literature constrains the design. Revisiting GRPO shows that stale sampling policies destabilize updates beyond a bounded drift. Our counter-exploiter is always a best response to a slightly stale policy, so we treat its refresh interval as a controlled variable. Advantage Collapse in GRPO and Revisiting GRPO disagree on whether to mask or re-anchor zero-variance groups. Poker has an analogous case: all rollouts fold preflop for the same loss. We simply measure how often it occurs rather than assume either remedy.

Research Program

The policy is a 7B instruction model. It receives a textual game state, may reason briefly, and emits one legal action. It is initialized by reproducing the PokerBench supervised recipe on its released solver spots, so it starts near equilibrium and near the documented passive bias.

In Leduc Hold'em (288 information sets) and 5-rank Leduc (780), we can read off the language model's complete action distribution by querying it at every information set. This gives the exact tabular policy it actually plays. From that policy we compute its exact best response $B_\pi$ and its exact exploitability. This setting gives ground truth for everything: the dual update uses true exploitability, and every reported number is exact.

In heads-up no-limit hold'em no exact audit exists. There, $B_\pi$ is a smaller learned attacker, trained with GRPO against frozen snapshots of the agent and kept as a growing population using the Pipeline PSRO codebase. Its win rate is a lower bound on true exploitability. We will report no-limit results as robustness against the strongest attacker we can train, never as a certificate.

Opponent models $M$ are scripted exploitable styles (over-folder, calling station, maniac) plus the PokerBench-style passive supervised model. The work is an offline question: given an opponent model, train a policy that exploits it within a budget. Online detection of opponents, and the confirmation-starvation problem CS-RNR raises, are out of scope.

Baselines

The naive reference points are the two ends of the tradeoff. The first is the supervised solver-imitation model alone, which is roughly unexploitable in intent but earns little. The second is plain GRPO trained only against $M$, which should earn the most against $M$ and be highly exploitable. Every other method must land between them. The frontier, winnings against $M$ as a function of exploitability, is the primary object of comparison.

The strongest existing method in Leduc is CS-RNR itself, run with its own solver at the same budgets $\epsilon$. Its reported points are zero certificate violations, 6.2× a binary gate's gain, and 0.15 exploitability where the mixture reaches 2.04. We treat it as an upper reference, since an exact tabular solve should dominate a learned policy in a game this small. The fair learned-policy version of this baseline is CS-RNR distilled: supervised fine-tuning of the same 7B model on CS-RNR's restricted-response strategy, with the same number of gradient steps and sampled tokens as our method. We beat it if our realized-policy frontier lies above the distilled model's realized-policy frontier. "Realized" means exploitability measured on the policy the language model actually plays, not on the strategy it was trained to copy.

In no-limit hold'em the strongest external reference is PokerBench: 78.26% exact match for the fine-tuned model and GPT-4's 22.20 bb/100 margin over it. No reported result found on Lacuna for any RL-trained language model in no-limit hold'em under exploitability constraints. We beat PokerBench if our model wins against GPT-4 in heads-up play under PokerBench's protocol, and at the same time loses less to our trained attackers than the supervised model does.

Two ablations isolate the key ingredient. The first pools all $G$ rollouts into one normalization instead of standardizing per opponent. The second applies naive reward-minus-λ-cost scalarization with the same dual update. Constrained GRPO predicts both should miss the budget. If pooled normalization matches the stratified version, our central technical claim is wrong. A third ablation fixes $\lambda$ instead of adapting it, to test whether the dual update is what hits $\epsilon$.

All methods share base model, data, rollout count, and a matched hyperparameter search of equal size.

Why not just…?

Why not just distill CS-RNR into the language model? In Leduc this is a real threat, and it may match us. If it does, we will say so. Our method's value then rests entirely on games where no tabular solve exists. The argument against distillation is that imitation controls actions at training states, not the exploitability of the resulting policy. PokerBench shows high exact-match accuracy coexisting with heavy exploitation. The distillation baseline above measures this gap directly in the one setting where it can be measured exactly.

Why not just mix an equilibrium model and an exploiting model at play time? CS-RNR reports that such trajectory mixtures reach exploitability 2.04 against a 0.15 budget in Leduc, because an adversary can target the exploiting branch across decisions. We include the mixture of our two naive endpoints to check that this finding transfers to language-model policies.

Why not just add an exploitability penalty to the reward in plain GRPO? There is no per-trajectory exploitability to subtract unless you change the opponent, which is our proposal. Even with that change, Constrained GRPO proves that joint normalization distorts the multiplier. The pooled and naive-scalarization ablations test whether that distortion matters here.

Why not just switch policies online, as SWAP does? Poker rewards arrive at showdown, and SWAP's authors note that bandit switching fails under sparse feedback. We run SWAP over our two endpoint policies, with per-hand reward as its signal, as an additional comparison in no-limit hold'em.

Proposed Experiments

In Leduc and 5-rank Leduc we sweep $\epsilon$ over a range bracketing CS-RNR's 0.15 budget. We plot exact realized exploitability against winnings versus each $M$ for all methods, over three seeds. The first decisive check is whether the stratified dual method keeps realized exploitability within $\epsilon$ at every checkpoint after warm-up, while pooled normalization overshoots. The second is whether its frontier sits above distilled CS-RNR.

We also vary the counter-exploiter refresh interval to find where staleness breaks the constraint, the analog of the drift limit in Revisiting GRPO. We log the fraction of zero-variance groups in each opponent half.

In heads-up no-limit hold'em we report four things. The first is winnings in bb/100 against each scripted $M$. The second is losses against a held-out attacker population trained after the fact, never seen during training, which guards against overfitting to the training attacker. The third is live heads-up play against GPT-4 under PokerBench's setup. The fourth is PokerBench exact match, to check that equilibrium knowledge is not destroyed.

The outcome that would falsify the proposal is this: in no-limit hold'em, held-out attackers beat our constrained model by as much as they beat plain GRPO against $M$. That would mean the learned counter-exploiter was too weak to stand in for a best response. Only the Leduc results would then survive.