Robust Estimators for Counterfactual Policy Evaluation

This direction is merged into another direction: off-policy bias-variance tradeoff. Its last version stays readable and citable here.

Version 1 of 153 papers56 conceptsfrozen 2026-10-04

In many real-world systems—from medical treatment protocols to e-commerce recommendation engines—we want to know if a new strategy will perform better than our current one. However, testing a new "target" policy live is often too risky, expensive, or unethical.

This creates the central challenge of Off-Policy Evaluation (OPE): how do we use historical logs generated by an old "behavior" policy to accurately predict the performance of a new, untested policy? The research in this field focuses on navigating a fundamental tension between two types of errors: the bias of our models and the variance of our data.

The Counterfactual Dilemma

To evaluate a policy without running it, we must answer a counterfactual question: "What would have happened if we had taken action A instead of action B?" As Maarten de Rijke notes in Don't hurt them: Learning to rank from historical interaction data, this is essential for protecting user experience while innovating.

There are two classical ways to approach this, each with a fatal flaw:

  1. The Direct Method (DM): We build a reward model (a simulator) of the environment. We then "run" our new policy through this model. This is stable and has low variance, but if our model is even slightly wrong—a state called model misspecification—our evaluation will be systematically biased.
  2. Importance Sampling (IS): We don't model the world; we simply re-weight the rewards we actually saw in the past. If the target policy was more likely to take an action than the behavior policy was, we scale that reward up. This is statistically "unbiased," but the weights can explode. If the two policies disagree significantly, the variance becomes so high that the estimate becomes useless noise.

Bridging the Gap with Doubly Robust Methods

The modern standard for OPE is the Doubly Robust (DR) estimator. The intuition is to use the reward model as a baseline and use importance sampling only to correct the model's errors.

If the reward model is perfect, the IS correction term averages to zero, and we get a low-variance estimate. If the model is totally wrong, the IS term (in expectation) corrects the bias. As Nathan Kallus and Masatoshi Uehara demonstrate in Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning, we can design estimators like REG and EMP that are "intrinsically efficient," meaning they are guaranteed to perform as well as or better than existing methods even when the underlying models are misspecified.

To further refine this, Hao Liu and Anima Anandkumar proposed "Triply Robust" estimators in Triply Robust Off-Policy Evaluation, which add a third layer of protection by accounting for covariate shift—the change in the distribution of states the new policy might visit.

Taming the Curse of Horizon

In reinforcement learning, the problem is compounded by time. In a long sequence of actions, the importance weights for each step are multiplied together. This leads to the "curse of horizon," where variance grows exponentially with the length of the task.

Tengyang Xie and Yu-Xiang Wang addressed this in Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling. Instead of weighting entire trajectories, Marginalized Importance Sampling (MIS) estimates the ratio of how often the target policy visits a specific state compared to the behavior policy. This reduces the variance from exponential to polynomial, making long-term evaluation feasible.

Adaptive Blending and Shrinkage

Even with these tools, there is no "one size fits all" estimator. Sometimes the data is so sparse that we should trust a biased model over a high-variance weight.

Researchers like Yi Su and Thorsten Joachims introduced Continuous Adaptive Blending (CAB) in CAB: Continuous Adaptive Blending for Policy Evaluation and Learning. CAB dynamically decides how much to trust the importance weights based on their magnitude. Similarly, Akshay Krishnamurthy and Miroslav Dudík developed "shrinkage" techniques in Doubly robust off-policy evaluation with shrinkage, which systematically cap weights to find the optimal point on the bias-variance frontier.

Validation and Real-World Impact

How do we know if our OPE estimator is working if we don't have the ground truth? Branislav Kveton and colleagues proposed Off-Policy Cross-Validation (OCV) in Cross-Validated Off-Policy Evaluation, using held-out data to select the best estimator and hyperparameters.

To ground this research, Yuta Saito and Yusuke Narita released the Open Bandit Dataset, providing millions of real-world data points from the ZOZOTOWN platform. This allows researchers to compare OPE estimates against actual A/B test results, ensuring that the mathematical progress in "intrinsically efficient" and "triply robust" estimators translates into reliable decision-making in the real world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_robust_estimators_for_counterfactual_pol_430bd58c,
  title = {Robust Estimators for Counterfactual Policy Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-estimators-for-counterfactual-policy-evaluation/rdr_430bd58ce0a14f33a9e08ae06aa7ddbe},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_robust_estimators_for_counterfactual_pol_430bd58c_v1,
  title = {Robust Estimators for Counterfactual Policy Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-estimators-for-counterfactual-policy-evaluation/rdr_430bd58ce0a14f33a9e08ae06aa7ddbe/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}