Variational Approach to Optimal IPS Estimator for Multi-logger Off-Policy Evaluation

Joon Suk Huh, Junghoon SeoarXiv 2026 · arXiv

In the world of data-driven decision-making, Off-Policy Evaluation (OPE) is a critical tool. It allows us to estimate how a new "evaluation" policy (like a new recommendation algorithm) would perform using only historical data collected by "logging" policies. But what happens when that data comes from multiple sources—say, different versions of a system running in parallel or sequential updates to a production model?

A new paper from researchers at the University of Wisconsin–Madison and PIT IN Corp. introduces a mathematically elegant solution to this "multi-logger" problem using variational calculus.

TL;DR

The authors derive a new estimator ($J_{oIPS}$) that finds the mathematically optimal way to weight data from multiple logging policies to minimize variance. By treating the weighting problem as a variational optimization task, they create an estimator that is more robust and accurate than existing methods, even when the underlying reward models are imperfect.

What Problem Does This Paper Address?

OPE typically relies on Inverse Propensity Scoring (IPS), which reweights observed rewards based on the ratio between the new policy and the old one. In the multi-logger setting, you have a choice: do you treat all samples equally (Naive IPS), use an average of the logging policies (Balanced IPS), or weight each logger by its estimated variance (Weighted IPS)?

As it turns out, there is no "one size fits all" answer. Previous research showed that Balanced IPS and Weighted IPS trade blows depending on the specific dataset. This "instance-dependence" creates a headache for practitioners who don't know which method to trust. The challenge is to find a single estimator that adapts to the data and achieves the lowest possible variance without introducing bias.

Methods

The authors move away from choosing between fixed weighting rules and instead define a broad class of generalized weighted-IPS estimators. They then use variational calculus to solve for the weight functions that minimize the estimator's variance subject to an unbiasedness constraint.

Key Innovations:

  • The Technique Matrix ($T$): Instead of needing a complex reward model (which can be hard to train), their estimator only requires a $K \times K$ matrix (where $K$ is the number of loggers) and a $K$-vector of moments. This makes the method computationally efficient.
  • Control-Variate Form: The resulting estimator can be viewed as a standard balanced-IPS estimator augmented with a "control variate." This extra term uses the known logging propensities to "soak up" noise and reduce variance.
  • Geometric Projection: They show that their method essentially projects the reward-weighted evaluation policy onto the space spanned by the logging policies. The "gap" in this projection directly corresponds to the extra variance compared to a theoretical "oracle" estimator.
  • Doubly Robust Extension ($J_{DR-oIPS}$): They extend the logic to a Doubly Robust framework, which combines the IPS approach with a learned reward model for even greater stability.

Datasets

The researchers evaluated their approach using:

  1. UCI Benchmark Datasets: OptDigits, SatImage, PenDigits, and Letter. These classification tasks were converted into contextual bandit problems to simulate real-world OPE scenarios.
  2. Synthetic Data: Large-scale experiments ($N$ up to $10^6$) using the Open Bandit Pipeline (OBP) to test asymptotic behavior and computational scalability.

Key Insights and Ideas

  • Resolving the Dilemma: The proposed $J_{oIPS}$ is proven to be at least as efficient as Naive, Balanced, and Weighted IPS in the limit. It effectively "solves" the choice problem by adapting its weights to the specific overlap between the evaluation and logging policies.
  • Robustness to Misspecification: One of the most striking "aha moments" is that their Doubly Robust version ($J_{DR-oIPS}$) can remain optimal even if the reward model is wrong. As long as the error in the reward model can be represented by the logging policies, the estimator corrects for it automatically.
  • Reward-Model-Free Efficiency: The paper demonstrates that you can get very close to the theoretical "efficiency bound" (the best possible performance) using only the logging policies, without ever actually modeling the rewards directly.

Key Results

  • Consistent Wins: Across all benchmark datasets and various mixtures of loggers, $J_{oIPS}$ consistently outperformed non-DR baselines.
  • Superior Stability: In many settings, the new DR estimator ($J_{DR-oIPS}$) achieved lower Relative-RMSE than the previous state-of-the-art baseline ($J_{DR-bIPS}$ from Kallus et al., 2021).
  • Concrete Gains: In synthetic experiments with a million samples, $J_{oIPS}$ achieved a Relative-RMSE of $1.99 \times 10^{-4}$, significantly lower than the $5.44 \times 10^{-4}$ achieved by Weighted IPS.
  • Scalability: Even with 100 different logging policies, the overhead of solving the $K \times K$ linear system was minimal (about 20ms per trial), making it practical for real-world production pipelines.

Limitations and Future Work

  • Known Propensities: The current theoretical guarantees assume the logging policies are known. While the authors provide a version for estimated propensities, the variance properties change and require further study.
  • Overlap Requirements: Like all IPS-based methods, this approach relies on "sufficient overlap"—if the logging policies never explore actions the evaluation policy wants to take, the estimate will be unreliable.
  • Beyond Bandits: The current work focuses on contextual bandits (single-step decisions). Extending this variational approach to full Reinforcement Learning (multi-step trajectories) is a promising but complex next step.

For more details, you can find the full paper under the title "Variational Approach to Optimal IPS Estimator for Multi-logger Off-Policy Evaluation" in the NeurIPS 2026 proceedings.

Summary updated

Abstract

We study off-policy evaluation (OPE) in contextual bandits with data collected from multiple logging policies. Inverse propensity scoring (IPS) is a standard approach to OPE and extends naturally to the multi-logger setting. However, as highlighted by Agarwal et al. [2017], neither balanced nor weighted IPS consistently outperforms the other in this setting. We resolve this dilemma by deriving an optimal IPS estimator with sample-dependent weights that minimize variance subject to unbiasedness. Using a variational calculus approach, we obtain closed-form optimal weights, yielding an estimator that is unbiased and achieves asymptotically optimal variance within a generalized weighted-IPS estimator class. Instead of a reward model, the estimator needs only a $K\times K$ matrix and a $K$-vector of moments, where $K$ is the number of logging policies, and we quantify how much efficiency this costs relative to the semiparametric oracle. Experiments on benchmark datasets show that our estimator generally improves on existing multi-logger IPS methods. We also extend our estimator to a doubly robust form by incorporating a conditional reward estimator. We compare the resulting DR extension with the semiparametrically efficient DR estimator of Kallus et al. [2021], both theoretically and empirically. We show that the two estimators achieve the same asymptotic variance when the conditional reward estimator is consistent, while our estimator can retain semiparametric efficiency under certain forms of reward misspecification. Empirically, the proposed DR estimator often achieves lower estimation error than the existing DR baseline.
BibTeX
@misc{huh2026variational,
  author = {Joon Suk Huh and Junghoon Seo},
  title = {{Variational Approach to Optimal IPS Estimator for Multi-logger Off-Policy Evaluation}},
  year = {2026},
  eprint = {2611.11712},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11712},
}
Figure