Catastrophic Overfitting in Cheap Minimax Off-Policy Evaluation

When an offline dataset was collected by decision-makers who saw something the dataset does not record, such as a clinician's read of how sick a patient looks, standard methods for estimating how a new policy would perform are biased. More data does not remove that bias. The main fix for sequential problems learns "bridge functions," which are models of observable proxies that stand in for the hidden variable. These are trained through a two-player game: one network fits the bridge, and a second network, called the discriminator, searches for the observable test function on which the fit fails. That game is expensive and unstable. The adversarial-robustness literature has spent years making the same kind of inner maximization cheap, and it found a sharp failure mode in the process. Cheap single-step inner solvers can produce models that look robust to the weak adversary used in training while failing completely against a strong one. This is called catastrophic overfitting.

This note proposes to bring both halves of that experience into confounded off-policy evaluation. We first test whether cutting the cost of the discriminator loop produces the same kind of hidden failure: a bridge function that satisfies its training discriminator but badly violates the moment conditions that a fresh, strong discriminator can find, which silently biases the estimated policy value. Second, we propose a cheap repair and a cheap check. The repair is a "free" discriminator update scheme with random restarts and replay, adapted from fast adversarial training. The check is a post-hoc audit discriminator. It needs only observed data and no ground truth, so it also works on real clinical logs.

Core Claim

The bridge-learning estimators of A Minimax Learning Approach to Off-Policy Evaluation in Confounded POMDPs fit a bridge $b$ so that a conditional moment holds. In schematic form,

$$\min_b \max_{f \in \mathcal{F}} \; \mathbb{E}\big[(Y - b(W,A))\, f(Z,A)\big] - \lambda \|f\|^2,$$

where $W$ and $Z$ are observed proxies (for example, recent and earlier history), $A$ is the action, $Y$ is the outcome target, $f$ is the discriminator drawn from class $\mathcal{F}$, and $\lambda$ is a regularization weight.

This is structurally the same bi-level problem as adversarial training, with $f$ playing the attacker. We claim three things.

First, giving the discriminator fewer inner steps produces a sudden collapse of bias, not a gradual one, in the way FGSM training collapses. The training-time moment violation stays small while the true policy-value error jumps.

Second, an audit statistic $V(b) = \max_{f \in \mathcal{F}'} \hat{\mathbb{E}}_{\text{held-out}}[(Y - b)f] - \lambda\|f\|^2$ tracks true error closely. Here $\mathcal{F}'$ is a richer class, re-initialized and trained to convergence against the frozen $b$. This makes the audit the analogue of PGD evaluation after FGSM training.

Third, a free-update scheme recovers the accuracy of a full $K$-step alternating solver at close to the cost of one step. In this scheme the bridge and discriminator share each backward pass, the discriminator persists across minibatches with periodic random re-initialization, and a small replay buffer holds past discriminators.

The first claim is what would be new. The literature says that minimax OPE is "unstable" (Minimax Weight and Q-Function Learning; Shi et al.). Nobody has characterized the failure as masking of a weak discriminator, or given a ground-truth-free way to detect it.

Relation to Prior Work

The motivating work is Shi et al.'s confounded-POMDP estimators. They introduce value and weight bridges learned by minimax and report sensitivity to discriminator classes and hyperparameters. We keep their estimators unchanged and study only the inner solver. Uehara et al.'s MWL/MQL apply the same game to unconfounded problems and serve as a reference method.

The method we borrow comes from fast adversarial training. Adversarial training for free! recycles one backward pass to update both the model and the perturbation. Work on catastrophic overfitting (A Training-Dynamics View of Catastrophic Overfitting, Understanding and Improving Fast Adversarial Training, Regularizer to Mitigate Gradient Masking) shows that random initialization and anti-masking regularization prevent collapse. We translate random initialization into discriminator restarts, and we translate multi-step evaluation into the audit.

Two lines of work constrain the design. Efficient and Sharp Off-Policy Learning under Unobserved Confounding removes minimax entirely through a closed-form bound, but only for static, discrete-action problems under a sensitivity model. Our target is point-identified sequential problems with proxies, where no closed form is known. Kernel methods (Optimal Balance, Kernel Conditional Moment Constraints) also avoid deep games, but they report scaling bottlenecks. Adversarial De-confounding in ITE Estimation uses gradient reversal to purge treatment information from a learned representation. It assumes all confounders are observed and does not study the cost of its adversary, so it neither covers nor contradicts our question.

Research Program

We will implement Shi et al.'s value-based, weight-based, and doubly robust estimators with neural bridges and neural discriminators. We then vary only the inner solver along a compute axis, from $K$ alternating discriminator steps per bridge step ($K \in \{1, 2, 5, 10, 20\}$) to the free scheme. For each run we log four quantities against cumulative gradient evaluations: the true policy-value error, which is available in simulators; the training discriminator's moment violation; the audit violation $V(b)$; and wall-clock time.

The analysis asks three questions. Does error at low $K$ show bimodal, run-dependent collapse, rather than a smooth degradation? Does collapse coincide with a gap between the training violation and the audit violation? Does the doubly robust estimator hide the failure? It might, if one bridge collapses and the other compensates, so we measure each bridge's audit separately.

The free scheme will then be tuned under the same budget as the baselines. Its ablations remove restarts, remove replay, and remove the shared backward pass.

Baselines

The naive reference treats observations as the full state and runs fitted Q-evaluation without any bridge. On Shi et al.'s confounded toy problem, naive methods missed the true value by "nearly 40%" (Shi et al., reported as relative error). This figure anchors how much bias is at stake.

The unconfounded minimax reference is MQL/MWL. It isolates the value of modeling confounding at all.

The strongest existing method is Shi et al.'s doubly robust estimator with a full alternating solver at large $K$, tuned per environment. We could find no specific MSE value for it on Lacuna; the file reports only that it outperformed minimax Q-learning across noise levels and masks. We will therefore re-run it ourselves on identical data with the same tuning budget. Beating it means matching its policy-value MSE within seed-level confidence intervals at no more than one-third of its gradient evaluations, and with lower variance across seeds.

The second strong competitor is a closed-form inner maximum. We replace the neural discriminator with a random-feature kernel class, for which the regularized maximum over $f$ has an analytic solution per minibatch. This removes the game in the same spirit as the closed-form and kernel lines of work.

The key ablation is the free scheme without restarts. If restarts are what prevents collapse, as random initialization does in fast adversarial training, this ablation should fail in the same runs where low-$K$ fails.

All methods share the same data, network sizes for the bridges, seeds, and a fixed hyperparameter-search budget measured in GPU hours.

Why not just…?

Why not just use a closed-form kernel discriminator and skip the game? This is the real threat. With random features the inner maximum is analytic and cheap, and on Shi et al.'s low-dimensional 1D process it may simply win. We run it as a main baseline and expect it to be competitive there. The open question is high-dimensional proxies: masked CartPole histories and sepsis features, where a fixed feature map may miss the directions in which the bridge fails. The audit discriminator settles this, because it can be run against kernel-trained bridges too. If kernel bridges pass the audit at matched cost in every setting, we will report that and shift the contribution to the diagnostic alone.

Why not just run more discriminator steps? That is the expensive reference, and it may well be the most accurate. The point of the study is to map how much of that cost is necessary. Without an audit, practitioners cannot tell from training curves whether their $K$ was enough, because the training violation is exactly the quantity that masking makes look healthy.

Why not use sensitivity bounds with closed-form sharp solutions instead of bridges? Those methods answer a different question. They give worst-case intervals under an assumed confounding strength, and they are derived for static discrete actions (Efficient and Sharp Off-Policy Learning). When valid proxies exist, bridges give point estimates. Persistent confounding across steps makes sensitivity intervals widen exponentially with horizon (Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding).

Proposed Experiments

We use the 1D dynamic process and the modified CartPole with observation masks and noise from Shi et al., the confounded GridWorld from Off-policy Evaluation in Infinite-Horizon RL with Latent Confounders, and the sepsis simulator from Kallus et al. The sepsis simulator is the closest to practice, and on it we measure whether audit rankings agree with true-error rankings.

The first decisive check is the collapse test. Across at least 20 seeds per $K$, low-$K$ runs should show a heavy-tailed or bimodal error distribution, and collapsed runs should show a training-versus-audit violation gap. If error instead degrades smoothly and the training violation alone predicts it, the catastrophic-overfitting analogy is wrong and we say so.

The second decisive check is audit validity. Across all runs, the rank correlation between $V(b)$ and true policy-value error should be high. Its correlation should also be clearly higher than that of the training violation.

The third decisive check is the cost frontier. The free scheme with restarts should sit on or near the accuracy of the large-$K$ solver at a fraction of its gradient evaluations, and the no-restart ablation should not.