Quantifying Uncertainty in Offline Policy Evaluation

34 concepts29 papers1 proposal

In reinforcement learning and causal inference, evaluating a new policy using historical data—known as Off-Policy Evaluation (OPE)—is a foundational challenge. Traditionally, OPE algorithms focus on producing a single point estimate of a policy's expected return.

However, in high-stakes domains like healthcare, finance, or autonomous systems, relying on a single number is dangerous. Finite datasets, distribution shifts, and unobserved environmental factors introduce statistical noise and bias. If an evaluation algorithm predicts a high average return but hides a massive variance, deploying that policy could result in catastrophic failure.

To bridge this gap, a major research direction focuses on policy value uncertainty quantification. Instead of asking "What is the expected value of this policy?", this field asks "What is the statistically safe lower bound of this policy's performance?" By moving from point estimation to rigorous statistical inference, researchers are developing tools to construct reliable confidence intervals, variance estimates, and worst-case bounds.


The Core Challenge: Why Point Estimates Fail

When evaluating a target policy $\pi$ using data generated by a different behavior policy $\mu$, two primary sources of uncertainty emerge:

  1. Epistemic Uncertainty: We only observe a finite number of trajectories. Rare but catastrophic outcomes might be missing from our dataset entirely.
  2. Distribution Shift: The target policy $\pi$ visits states and takes actions with different frequencies than the behavior policy $\mu$. This amplifies variance, especially when computing importance sampling weights over long horizons.

As noted by Ziyang Tang et al. and Bo Dai et al. (2020), point estimates completely obscure these risks. If an algorithm cannot quantify its own limitations, practitioners cannot guarantee that a new policy will meet minimum safety thresholds. As Munther A. Dahleh argues in a broader data science context, predictive algorithms are insufficient for high-stakes decision-making unless they provide a clear assessment of their potential for error in complex environments.


Mechanism 1: Leveraging Asymptotic Normality and Resampling

One of the most direct ways to build confidence intervals is to exploit the asymptotic properties of policy value estimators. If an estimator behaves predictably as the dataset grows, we can compute its variance and construct traditional statistical bounds.

Asymptotic Normality and Taylor Expansions

For complex environments like Partially Observable Markov Decision Processes (POMDPs), finding exact distributions is difficult. Mahdi Milani Fard et al. address this by applying a second-order Taylor expansion to the policy's value function. This analytical approach derives explicit expressions for bias and variance when transition and observation models are estimated from finite data, bypassing the need for expensive Monte Carlo simulations.

In infinite-horizon settings, Chengchun Shi et al. (2021) establish the asymptotic normality of value function estimators, allowing practitioners to determine if a policy's performance represents a statistically significant improvement rather than random noise. Similarly, in the context of Individualized Treatment Regimes (ITRs), Minxin Lu et al. leverage this asymptotic normality to construct computationally efficient $t$-tests to compare different personalized treatment rules without relying on heavy resampling.

Distributional Consistency via Bootstrapping

When analytical variance is too difficult to compute directly, bootstrapping offers a non-parametric alternative. However, standard transition-level bootstrapping fails in sequential decision-making due to temporal dependencies.

To fix this, Botao Hao et al. provide a theoretical foundation for bootstrapping Fitted Q-Evaluation (FQE). By resampling data at the episode level rather than the transition level, the authors prove that the resulting estimator is asymptotically efficient and distributionally consistent. This allows practitioners to map out the true distribution of the estimator and extract tight, mathematically justified confidence intervals.


Mechanism 2: Non-Asymptotic Bounds via Empirical Likelihood

While asymptotic methods work well with large datasets, they can fail catastrophically when data is scarce, producing intervals that are either overly optimistic or excessively conservative. To obtain guarantees that hold for finite samples, researchers turn to optimization frameworks like empirical likelihood.

Instead of assuming a specific distribution (like a Gaussian), empirical likelihood methods find the most pessimistic and optimistic data distributions that are still statistically plausible given the observed data.

   [ Observed Data Distribution ]
                 │
        ┌────────┴────────┐
        ▼                 ▼
[ Pessimistic Weights ] [ Optimistic Weights ]
        │                 │
        ▼                 ▼
  Lower Bound       Upper Bound
  (Safe Deployment) (Exploration Potential)

For instance, the CoinDICE algorithm introduced by Bo Dai et al. (2020) frames off-policy confidence interval estimation as a minimax optimization problem. By optimizing over the space of stationary distribution corrections, it computes tight frequentist intervals even when the behavior policy is completely unknown.

This optimization approach is highly adaptable:

  • Cost Constraints: Nicholas Polosky et al. adapt empirical likelihood to Constrained Offline Policy Optimization (COPO), providing non-asymptotic upper bounds on policy costs to ensure safety limits are never breached.
  • Bypassing Importance Sampling: To avoid the high variance and "curse of horizon" typical of standard importance sampling, Yihao Feng et al. introduce a variational framework using Kernel Bellman Statistics, offering behavior-agnostic confidence intervals by minimizing a kernelized regularized loss.
  • Finite-Sample Model Bounds: For average-reward MDPs, Nevena Lazic et al. utilize a Maximum-Entropy approach to derive the first finite-sample error bounds for model-based OPE in linear environments.

Mechanism 3: Bounding the Unobservable (Confounding and Shifts)

A particularly thorny problem arises when the assumption of "no unobserved confounding" is violated—meaning hidden variables influenced both the historical actions and the observed rewards. In these scenarios, the true value of a policy is fundamentally unidentifiable; the data alone cannot pinpoint a single expected return.

Rather than giving up, researchers focus on deriving sharp bounds—the tightest possible interval of values consistent with the observed data under a specified level of environmental uncertainty or confounding sensitivity.

Robust MDPs and Sensitivity Analysis

In infinite-horizon settings with unobserved confounders, Nathan Kallus et al. demonstrate how to compute sharp bounds that define the exact envelope of possible policy values.

The core challenge is preventing these bounds from becoming too wide to be practically useful. David Bruns-Smith refines this by distinguishing between hidden variables that change at every step versus those that persist over time, creating a tighter Robust MDP framework. Similarly, Hongseok Namkoong et al. exploit the specific structure of single-decision confounding in sequential environments to compute worst-case bounds efficiently, ensuring that an apparent policy improvement isn't merely an artifact of hidden bias.

This robust framing also applies to environment shifts (where the deployment environment differs from the data collection environment). Andrew Bennett et al. introduce an estimator for Robust MDPs that achieves both statistical efficiency and sharpness, ensuring that worst-case and best-case scenarios are accurately reflected without being overly conservative.


From Evaluation to Risk-Aware Policy Selection

Quantifying uncertainty is not just an academic exercise; it fundamentally changes how we select and deploy models. When presented with multiple candidate policies, picking the one with the highest estimated average reward can lead to overfitting and failure.

Instead, practitioners can use uncertainty estimates to implement Pessimistic Model Selection (PMS). As developed by Chao-Han Huck Yang et al., PMS uses asymptotic variance analysis to calculate a Lower Confidence Bound (LCB) for each model, selecting the policy with the highest worst-case performance.

Policy A: ───[=======( Point Estimate )=======]───  (High Variance, Risky)
Policy B:         ├───[==( Point Estimate )==]───┤  (Low Variance, Safe LCB)

Result: Pessimistic Model Selection chooses Policy B due to its higher safe lower bound.

For more complex selection criteria, Bo Dai et al. (2022) introduce BayesDICE, a Bayesian framework that generates the full posterior belief distribution of a policy's value. This allows decision-makers to optimize for specific real-world objectives, such as minimizing regret or maximizing top-$k$ precision, providing a robust foundation for deploying autonomous systems with confidence.

Key Foundational Work

Estimating the Reliability of MDP Policies: a Confidence Interval Approach (2007) — contributed 3 concept elements to this direction.
A Variance Analysis for POMDP Policy Evaluation (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_uncertainty_quantification_non_sharp_unc_f5ed8827,
  title = {uncertainty quantification (non-sharp uncertainty quantification)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uncertainty-quantification-non-sharp-uncertainty-quantification/rdr_f5ed8827e69446768df7171e49766c0f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_uncertainty_quantification_non_sharp_unc_f5ed8827_v1,
  title = {uncertainty quantification (non-sharp uncertainty quantification)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uncertainty-quantification-non-sharp-uncertainty-quantification/rdr_f5ed8827e69446768df7171e49766c0f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake