Centered, Two-Sided Robust Estimation of Policy Value from Logged Data

When we judge a new decision policy using data logged by an old one, the standard approach reweights each logged reward by how much more likely the new policy was to take the logged action. This is importance weighting. The weights can be enormous, so a handful of samples can dominate the estimate. Every popular fix works by shrinking the weighted rewards toward zero: clipping large weights, smoothing them, or passing weighted costs through a logarithm. Shrinking toward zero is harmless when rewards are nonnegative and the policy's value sits near zero. It is a poor choice in two other cases. The first is when rewards are signed, as with profit minus cost or a clinical gain against a harm. The second is when rewards carry a large constant offset. In either case the bias from shrinkage grows with how far the true value is from zero, not with how noisy the data are. The strongest recent estimators also give confidence bounds on one side only.

This note proposes a different target. We estimate the robust center of the importance-weighted outcomes, using a smooth penalty that is symmetric and has bounded slope, in the way Holland's bidirectional risks penalize loss deviations above and below a learned location. The resulting estimator shrinks outliers toward the current estimate rather than toward zero. It damps extreme samples on both sides, needs only a finite second moment of the weighted outcome, and yields a two-sided confidence interval from a single estimator. The headline test is a reward-offset sweep. If the claim holds, the error of our estimator stays flat as a constant is added to all rewards, while the error of the zero-centered methods grows.

Core Claim

Write $z_i = w_i r_i$ for the importance-weighted reward of logged sample $i$, where $w_i = \pi_e(a_i|x_i)/\pi_b(a_i|x_i)$ is the ratio of target to logging probability and $r_i$ is the observed reward. The same construction applies to doubly robust pseudo-outcomes, which subtract a reward model's prediction before reweighting. We define the estimate $\hat\theta_\sigma(\pi_e)$ as the root of

$$ \sum_{i=1}^{n} \psi\!\left(\frac{z_i - \theta}{\sigma}\right) = 0, $$

where $\psi(u)=\arctan(u)$ is the derivative of Holland's deviation function $\rho(u)=u\arctan u-\tfrac12\log(1+u^2)$, and $\sigma>0$ is a scale. Because $|\psi|\le\pi/2$, no single sample can move the estimate by more than a bounded amount in either direction. As $\sigma\to\infty$ the estimator recovers the ordinary importance-weighted mean.

We then choose the scale to balance bias and spread, setting $\sigma \asymp \sqrt{n v/\log(2/\delta)}$, where $v$ is an upper bound on $\mathbb{E}[z^2]$ and $\delta$ is the allowed failure probability. Under that choice we aim to prove a two-sided deviation bound

$$ |\hat\theta_\sigma - V(\pi_e)| \le C\sqrt{v\log(2/\delta)/n} $$

holding with probability $1-\delta$, where $V(\pi_e)$ is the true value and $C$ is a constant. The square-root form is what bounded-influence location estimators give: the bias is of order $v/\sigma$ and the stochastic error is of order $\sigma\log(1/\delta)/n$, and this scale equalizes the two.

Two features separate this from existing work. First, shrinkage is centered at the learned $\theta$, so the bias depends on how dispersed the weighted outcomes are, not on how far the value sits from zero. Second, one estimator gives both a lower and an upper confidence bound, and the bounds allow signed, unbounded rewards.

Relation to Prior Work

The mechanism comes from Holland's bidirectional risks. Learning with risks based on M-location introduced the arctan-based deviation penalty and the learnable location. Flexible risk design using bi-directional dispersion generalized it to the Barron penalty family and noted that bounded-slope penalties act as implicit gradient clipping. Both papers use these penalties to shape the distribution of per-example supervised losses as a risk preference. We use the same functions for a different purpose: to estimate a mean from heavy-tailed importance-weighted pseudo-outcomes, whose center is the policy value. What transfers is the smooth, bounded-slope penalty and the location-plus-scale design, not the risk interpretation.

The motivating gap comes from the pessimistic off-policy literature. Logarithmic Smoothing derives an estimator of the form $-\frac1\lambda\log(1-\lambda w c)$ from concentration bounds. It achieves finite variance with unbounded weights, but it bounds only the upper side and requires costs in $[0,1]$. Structurally it is a one-sided, zero-centered bounded-influence transform, and our estimator is its centered, symmetric counterpart. Log-Sum-Exponential damps only large positive values, because its theory is limited to $\lambda<0$. The harmonic correction of Metelli et al. and the power-law smoothing of Exponential Smoothing both shrink weights toward zero.

Two papers constrain the design. The Exponential Smoothing paper argues that one-sided bounds let optimizers settle on degenerate estimators, which motivates our two-sided interval. The unified PAC-Bayesian study found clipping, implicit exploration, and exponential smoothing "nearly identical" when placed under one bound. That result warns us that differences among zero-centered regularizers may be cosmetic. Our claim is therefore not "a better smoother." It is that the centering point matters, and the offset experiment tests exactly that.

The Wasserstein concentration results of Bhat and Prashanth give two-sided bounds for risk measures. They also constrain the design, because they rely on sub-Gaussian tails. Importance-weighted outcomes do not have sub-Gaussian tails, so we do not take that route.

Research Program

The work has three parts.

The first part is theory. We will prove the two-sided bound for the arctan estimator applied to both importance-weighted and doubly robust pseudo-outcomes, with $v$ replaced by a data-dependent estimate of the second moment. We will also make the bias term explicit, showing that it scales with $\mathbb{E}[(z-V)^2]/\sigma$ rather than with $\mathbb{E}[z^2]/\sigma$ when the estimator is centered by a preliminary estimate.

The second part is scale selection, which remains an open "oracle gap" for every smoothing parameter in this literature. We will compare a plug-in rule using the empirical second moment against a split-sample rule that searches over a grid of scales.

The third part is policy learning. We will optimize the upper confidence bound on cost for linear Gaussian policies. We will also study whether the bounded slope of $\psi$ gives stable gradients when the true value is far from zero, the regime where zero-centered clipping gives flat or biased gradients.

Baselines

The naive reference is plain inverse propensity scoring, together with its doubly robust version, both as implemented in the Open Bandit Pipeline. Any method that fails to beat these under heavy-tailed weights is not worth pursuing.

The strongest evaluation competitor is doubly robust estimation with optimistic shrinkage from Su et al.. On the Open Bandit Dataset "All" campaign at $n=300{,}000$, it reports an RMSE of 0.316, a 34% improvement over standard doubly robust estimation (A Large-scale Open Dataset for Bandit Algorithms). It is the strongest competitor because its reward model already centers the outcomes, which partly addresses the offset problem we target. We will run it through the same pipeline, with the same reward model (gradient boosting, which that benchmark found strongly affects accuracy), the same logged samples, and its automatic tuning. Our scale will receive a tuning budget of matched size. Beating it means lower RMSE at matched $n$ on the Open Bandit Dataset. At minimum, our estimator applied to the same doubly robust pseudo-outcomes should never be worse than shrinkage and should be clearly better at small $n$ or with offset rewards.

The strongest confidence-bound competitor is logarithmic smoothing. On kropt at $n=2^{14}$ it reports a relative radius of 0.035, against 0.041 for implicit exploration (Logarithmic Smoothing). Because it is one-sided and needs costs in $[0,1]$, we will give it a two-sided form by mirroring it. We will apply it to affinely rescaled costs and to their negation, and combine the two bounds with a union bound. Beating it means narrower two-sided intervals at equal or better empirical coverage.

We will also run self-normalized importance weighting, harmonic correction, log-sum-exponential, and clipping, each tuned on the same grid budget. For learning, the baselines are clipping, exponential smoothing, and LS-LIN, trained under the unified study's protocol, for which no reported result was found on Lacuna in numeric form.

Two ablations isolate the key ingredient. The first freezes $\theta=0$, turning our method into a zero-centered bounded-influence shrinker. The second replaces the symmetric $\psi$ with a one-sided version that damps only one tail. If the frozen-center ablation matches the full method, centering is not what helps.

Why not just…?

Why not just use self-normalized importance weighting, which is already unaffected by reward offsets? This is a real threat on the offset axis. Self-normalization handles a constant shift, but it does not bound the influence of individual heavy-tailed samples, and it has no finite-sample two-sided bound under unbounded weights. The heavy-tailed Lomax and signed-reward experiments are designed to separate the two. If self-normalization plus clipping matches us there, we will report that the centering benefit is already available cheaply.

Why not just use a doubly robust reward model, which centers the outcomes? A good reward model does remove most of the offset. Our estimator therefore runs on doubly robust pseudo-outcomes rather than competing with them. The remaining question is whether robust centering adds value when the reward model is misspecified, which the adaptive M-estimation study shows is common. Comparing optimistic shrinkage against our estimator on identical pseudo-outcomes, while varying reward-model quality, settles this question.

Why not just mirror logarithmic smoothing to get two one-sided bounds? Mirroring gives two different point estimates and forces a rescaling of signed rewards into $[0,1]$. That rescaling reintroduces a zero-centered bias of the kind we are trying to remove. The interval-width comparison under the offset sweep measures exactly this cost.

Why not just use mean-variance penalization for dispersion? A variance penalty has an unbounded slope, of the form $2(L-\mu)$. Paired with exploding weights, that makes gradients worse, not better, as the Holland papers argue. We will include it only in the learning experiments, as a sanity check.

Proposed Experiments

The decisive experiment is the offset sweep. We take UCI supervised-to-bandit tasks (yeast, letter, kropt) and add a constant $b$ ranging from $-5$ to $5$ to every reward. We then track MSE, two-sided 95% coverage, and interval width for every estimator. The prediction is that error and width stay flat in $b$ for our estimator and for the full doubly robust methods, and grow with $|b|$ for clipping, harmonic correction, and both logarithmic smoothing and log-sum-exponential.

The second experiment uses a synthetic setting with two-sided heavy tails: signed Lomax rewards combined with near-zero logging propensities. Log-sum-exponential leads in the one-sided version of this setup, and the two-sided version tests whether its single-tail damping fails as predicted.

The third experiment is off-policy evaluation on the Open Bandit Dataset at $n=10{,}000$ and $n=300{,}000$, measured against the 0.316 RMSE reported for optimistic shrinkage.

The fourth experiment is policy learning on MNIST and FashionMNIST converted to signed-reward bandits, with rewards of $+1$ for a correct action and $-1$ otherwise. We report the true reward of the learned policy and the tightness of its certified bound.

The project stops making its main claim under a clear condition. If, across the offset sweep and the heavy-tail synthetic setting, the frozen-center ablation or self-normalization plus clipping comes within statistical noise of the full estimator, then centering is not the operative ingredient. We would then report the result as a negative finding that extends the parity observed in the unified study.