Statistical Inference for Off-Policy Evaluation

This direction is split into other directions: uncertainty quantification (non-sharp uncertainty quantification). Its last version stays readable and citable here.

Version 1 of 169 papers79 conceptsfrozen 2026-10-04

In high-stakes domains like medicine, finance, and autonomous driving, a single number is rarely enough to justify a decision. If a reinforcement learning (RL) agent suggests a new treatment for sepsis, a doctor doesn't just want to know the "expected" improvement; they need to know how certain that estimate is and what the worst-case scenario looks like.

Traditional RL focuses on finding the optimal policy—the best set of rules to follow. However, a growing body of research is shifting focus toward statistical inference: the ability to provide rigorous confidence intervals, p-values, and risk bounds for policies before they are ever deployed.

The Point Estimate Trap

Most off-policy evaluation (OPE) methods—which use historical data to predict how a new policy would perform—produce a point estimate. This is a single "best guess" of the policy's value. As noted by Chengchun Shi and Rui Song, point estimates are dangerously incomplete in high-stakes settings because they ignore the statistical noise inherent in finite datasets Statistical Inference of the Value Function for Reinforcement Learning in Infinite-Horizon Settings.

Without uncertainty quantification, a policy might appear successful simply due to a lucky sample of data. Peng Liao and Susan Murphy argue that providing statistically rigorous confidence intervals is essential to prevent the adoption of interventions that look good on paper but lack sufficient evidence of a true effect Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health.

Building the Bell Curve for Policies

To move beyond point estimates, researchers are deriving the asymptotic distributions of policy values. The goal is to prove that as we collect more data, our estimate of a policy's value will follow a predictable pattern—usually a Normal distribution (the Central Limit Theorem).

Yuting Wei and Alessandro Rinaldo have established these foundations for Temporal Difference (TD) learning, allowing practitioners to ask "how sure are we?" rather than just "what is the value?" Statistical Inference for Temporal Difference Learning with Linear Function Approximation. Similarly, Imon Banerjee and Diego Klabjan demonstrated that uncertainty in the environment's transition dynamics propagates predictably to the final value estimate, enabling principled uncertainty bounds Central Limit Theorems for Transition Probabilities of Controlled Markov Chains.

Bootstrapping the Unknown

When the mathematical formulas for variance are too complex—such as when using deep neural networks—researchers turn to bootstrapping. This involves repeatedly resampling the data to see how much the policy estimate fluctuates.

Hao Lu provided the first rigorous justification for bootstrapping in Fitted Q-Evaluation (FQE), proving that resampling at the episode level yields consistent and reliable confidence intervals Bootstrapping Fitted Q-Evaluation for Off-Policy Inference. For real-time applications, Pratik Ramprasad and Zhaoran Wang developed an online multiplier bootstrap that runs in parallel with the agent, allowing it to quantify its own uncertainty during the learning process Online Bootstrap Inference For Policy Evaluation in Reinforcement Learning.

Handling Bias and Confounding

A major hurdle in off-policy inference is that the historical data was often generated by a "behavior policy" (e.g., a doctor's intuition) that might be biased or influenced by unmeasured factors.

To address this, researchers use doubly robust and debiased estimators. Daniel J. Graham and Emma J. McCoy utilize Bayesian doubly robust estimators to account for the fact that interventions, like speed cameras, are often placed at "hotspots," which can skew simple evaluations Quantifying the causal effect of speed cameras on road traffic accidents via an approximate Bayesian doubly robust estimator.

For even more complex scenarios, Chengchun Shi and Victor Chernozhukov introduced "deeply-debiased" procedures. These methods iteratively reduce the bias introduced by slow-converging machine learning models, ensuring that the resulting confidence intervals remain valid even in high-dimensional spaces Deeply-Debiased Off-Policy Interval Estimation.

From Averages to Safety Guarantees

The ultimate goal of this research direction is to provide safety guarantees. Instead of reporting the average performance, methods like PoLeCe (Policy Learning with Confidence) by Adam Rosen and Liyang Sun provide a "reporting guarantee"—a mathematical promise that the policy's actual welfare will exceed a specific threshold with high confidence Policy Learning with Confidence.

Similarly, Longchao Da and Hua Wei use Approximate Bayesian Computation to estimate the full distribution of a policy's performance. This allows practitioners to rank policies based on their "worst-case" outcomes, identifying strategies that might have high average returns but carry a dangerous risk of catastrophic failure Probabilistic Offline Policy Ranking with Approximate Bayesian Computation.

By treating policy evaluation as a problem of statistical inference rather than just optimization, this field is building the necessary trust to move reinforcement learning from simulated environments into the real world.

Papers in this version

Showing 60 of 69 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_statistical_inference_for_off_policy_eva_29118136,
  title = {Statistical Inference for Off-Policy Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/statistical-inference-for-off-policy-evaluation/rdr_29118136804d4caa926f66c5601e3283},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_statistical_inference_for_off_policy_eva_29118136_v1,
  title = {Statistical Inference for Off-Policy Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/statistical-inference-for-off-policy-evaluation/rdr_29118136804d4caa926f66c5601e3283/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}