In offline reinforcement learning and contextual bandits, Off-Policy Evaluation (OPE) allows us to answer a crucial counterfactual question: How well would a new target policy perform if we deployed it, using only historical data generated by a different behavior policy?
This capability is vital in high-stakes domains like healthcare, dynamic pricing, and education, where running live experiments is risky or unethical. However, standard OPE methods rely heavily on the ignorability assumption (or sequential randomization assumption), which posits that the historical data records every single variable influencing both the chosen actions and the resulting rewards.
In practice, this assumption is frequently violated. Unobserved factors—such as a patient's unrecorded genetic predisposition, a sudden unlogged shift in market trends, or a consumer's unmeasured mood—simultaneously drive historical decisions and outcomes. This is the problem of unmeasured confounding. When it occurs, standard OPE estimators suffer from a persistent bias that does not disappear, even if we collect an infinite amount of data.
To make OPE reliable, a growing body of research focuses on breaking or bounding this hidden bias. The field attacks this problem through three main conceptual lenses: proxies, instrumental variables, and distributional robustness.
The Core Failure of Standard OPE
To understand why unmeasured confounding breaks traditional evaluation, consider standard Inverse Propensity Scoring (IPS). IPS weights each observed reward by the ratio of the target policy's action probability to the behavior policy's action probability:
$$\text{Weight} = \frac{\pi_{\text{target}}(A|X)}{\pi_{\text{behavior}}(A|X)}$$
This math assumes the behavior probability depends solely on the observed state $X$. But if a hidden confounder $U$ also influenced the historical choice, the true logging probability was $\pi_{\text{behavior}}(A|X, U)$. Because we only observe $X$, our calculated propensity is wrong.
For instance, if doctors historically prescribed a high-risk treatment primarily to patients with severe, unrecorded symptoms ($U$), the data will show poor outcomes for that treatment. A naive OPE algorithm will conclude the treatment is inherently bad, failing to realize that the historical policy selected it precisely because those patients were already at higher risk.
Strategy 1: Proxies and Bridge Functions
When we cannot observe the confounder $U$ directly, we can often observe proxy variables—features that do not directly cause the outcome but are strongly correlated with the hidden factor.
Imagine $U$ is a patient's hidden stress level. We cannot measure it, but we can measure blood pressure and heart rate ($Z$, the proxies). Instead of attempting to explicitly reconstruct the high-dimensional latent space of $U$, recent methods use mathematical "bridge functions" to project the latent information directly onto observable data.
[Hidden Confounder: U] ───> Affects both Actions & Rewards
│
▼
[Proxy Variables: Z] ───> Used by Bridge Functions to balance groups
In the context of contextual bandits, Bennett et al. introduced an "Optimal Balance" weighting approach. Rather than trying to model complex latent outcomes, their method directly optimizes weights on the observed data to match the distribution of proxies between the behavior and target policies, providing provable accuracy guarantees.
This idea extends naturally into sequential settings like Partially Observable Markov Decision Processes (POMDPs) and infinite-horizon reinforcement learning. Shi et al. developed a minimax learning framework that utilizes these bridge functions to evaluate policies in continuous environments without requiring full state observability. Similarly, Miao et al. framed the sequential OPE problem under non-parametric models as a sequential Non-Parametric Instrumental Variable (NPIV) problem, establishing the first finite-sample error bounds for continuous, confounded POMDPs.
Strategy 2: Instrumental Variables
Another powerful tool borrowed from econometrics is the Instrumental Variable (IV). An instrument is an observed variable $I$ that satisfies three conditions:
- It affects the chosen action $A$.
- It is independent of the unmeasured confounder $U$.
- It affects the reward $R$ only through its influence on the action $A$.
[Instrument: I] ───> [Action: A] ───> [Reward: R]
▲ ▲
│ │
[Hidden Confounder: U]
By observing how changes in the instrument $I$ alter the action $A$ and subsequently change the reward $R$, we can isolate the true causal impact of the action, effectively purging the bias introduced by $U$.
Xu et al. demonstrated how to leverage IVs for confounded OPE. They proved that while standard estimators exhibit a fixed bias that persists regardless of sample size, IV-based estimators achieve statistical consistency—meaning they are guaranteed to converge to the true policy value as the number of observed trajectories increases.
Strategy 3: Distributional Robustness and Sensitivity Analysis
What happens if we have no proxies and no instrumental variables? If the data offers no way to point-estimate the policy's value, we must pivot from finding a single answer to bounding the worst-case scenario. This is known as Sensitivity Analysis or Distributionally Robust Optimization (DRO).
Instead of assuming our observed data distribution is perfectly accurate, we assume the true distribution lies within an uncertainty set (often defined by a KL-divergence or $f$-divergence neighborhood) around our observed data. We then evaluate the policy against the worst-case distribution within that set.
- Worst-Case Environments: Si et al. introduced a DRO framework for offline contextual bandits that designs policies to perform robustly under environment shifts, utilizing a stable learning variant to prevent over-pessimism in data-sparse regions.
- Sharper Bounds: Ishikawa et al. advanced this by using kernel methods to approximate infinite-dimensional conditional moment constraints. This mathematical tightening yields significantly sharper (tighter) worst-case performance bounds than previous sensitivity models.
- Temporal Distinctions: In sequential settings, the nature of the confounder matters. Bruns-Smith developed a Robust MDP framework by distinguishing between hidden variables that change independently at every step (IID) versus those that persist over time. Accounting for this temporal structure prevents bounds from loosening excessively over long horizons.
- Out-of-Sample Guarantees: Moving beyond average performance, Ek et al. combined sample-splitting with a marginal sensitivity model to provide finite-sample guarantees for the entire distribution of potential losses, certifying a policy's safety floor even under model misspecification.
Strategy 4: Exploiting Structural Invariants
A final frontier in deconfounding relies on exploiting the structural properties of specific domains. If we know how the hidden variables enter the system, we can design architectures to isolate them.
For instance, in multi-agent systems or two-sided marketplaces (like ride-sharing), confounding can stem from unrecorded localized events. Yu et al. introduced the Two-way Deconfounder (TWD), which structures latent variables by assuming they are either specific to an individual agent or specific to a precise point in time. This structural factorization allows neural networks to "unmask" the hidden variables and prevent biased policy rankings.
In other scenarios, the complexity of the environment can be simplified by changing the abstraction level. Zhou et al. focused on "stateful contextual" settings (such as inventory management), demonstrating that if the physical state transitions are simple but the rewards depend on high-dimensional unobserved factors, the OPE problem can be reduced to training just two statistical models. Similarly, Majumdar et al. proposed grouping raw states into human-interpretable "concepts," which drastically reduces the high variance that typically plagues confounded OPE.
Finally, because unmeasured confounding introduces severe risks, researchers are shifting what they optimize for. Rather than maximizing the average expected reward, which might inherit dangerous hidden biases, Chen et al. developed a framework for quantile-optimal policy learning. By optimizing for a specific percentile (like the median or the bottom 10th percentile), they ensure a guaranteed safety floor, protecting decision-makers from catastrophic worst-case outcomes in confounded environments.
Key Foundational Work
Policy Evaluation with Latent Confounders via Optimal Balance (2019) — contributed 2 concept elements to this direction.
Distributionally Robust Policy Evaluation and Learning in Offline Contextual Bandits (2020) — contributed 1 concept elements to this direction.