Off-policy evaluation estimates how well a new decision policy would perform using only data logged by a different policy. A common way to put error bars on that estimate is to train several value networks and read the uncertainty off their disagreement. This note argues that these error bars share a limit with ensembles in general. Past a small number of members, disagreement stops growing. Because the members all learn from the same data, they tend to make the same mistake on states and actions the data never covered. Under distribution shift, the ensemble's interval therefore settles at a width smaller than the real error. Adding members cannot close that gap, since the missing uncertainty is error that every member shares.
The proposal has two parts. The first is to measure this ceiling directly on benchmarks where the true policy value is known. The second is to spend the same compute differently: build the ensemble from different kinds of evaluators and combine them with an explicit between-family variance term, so the interval covers the truth at the nominal rate. We accept that these intervals are deliberately non-sharp, meaning wider than the tightest possible but valid. The central claim is falsifiable. For target policies far from the logging data, coverage of same-architecture ensembles plateaus below nominal as ensemble size grows. The height of that plateau can be predicted, without ground truth, from how much different evaluator families disagree with each other.
Why this gap exists
The saturation literature explains why disagreement runs out, but only for supervised classification. When are ensembles really effective? shows that ensemble gains track the ratio of member disagreement to member error, with correlation R ≥ 0.96. It also shows that networks trained to near-zero training loss converge and start agreeing on their errors, while random forests keep disagreeing. On Power Laws in Deep Ensembles fits ensemble likelihood as $c + b n^{-a}$ in the ensemble size $n$. In that fit, $c$ is the infinite-ensemble ceiling and $b$ is the total available gain, which shrinks as the members get larger. Rethinking Fano's Inequality in Ensemble Learning attributes saturation to redundancy and to information lost when members are combined, but it covers only discrete labels. These papers motivate our mechanism. None of them studies intervals, continuous value estimates, or sequential data.
On the evaluation side, the tools that rely on ensembles do not examine how that reliance scales with ensemble size. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble finds that critic networks trained on the same transitions develop aligned gradients, so their disagreement on unseen actions collapses. Its fix, a penalty that pushes the members' action-gradients apart, is used for policy learning, and the coverage of the resulting ensemble spread is never checked. Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning cuts the required ensemble from 500 to 10 by keeping the learned policy near the data. That lever does not exist in evaluation, where the target policy is fixed and may be arbitrarily far from the data. Offline Policy Comparison with Confidence uses only model-based dynamics ensembles and does not vary ensemble size. It does find that continuous paired intervals beat majority voting, and we build on that finding. The closest statistical tools are Bootstrapping Fitted Q-Evaluation for Off-Policy Inference and Pessimistic Model Selection for Offline Deep Reinforcement Learning. The first resamples whole episodes and is consistent under linear function approximation, but it under-covers below 50 episodes. The second splits the data into chunks and ranks candidate policies by their lower confidence bounds. Both produce diversity only by varying the data given to one estimator family. Neither asks whether that kind of diversity can reach the error that all members share.
One result constrains the design. On Local Posterior Structure in Deep Ensembles shows that adding local Bayesian posterior samples to ensembles of 10 or more members hurts calibration, because the extra variance is spread too widely in weight space. We therefore do not add diversity by sampling weights near each member. Diversity must come from different inductive biases.
Core Claim
Consider an ensemble of $m$ same-family evaluators with individual estimates $\hat V_1,\dots,\hat V_m$ of the target policy's value $V^\pi$. Each member's error splits into a component shared by all members and a component specific to that member. The empirical spread estimates only the member-specific variance $\sigma_w^2$, so the spread approaches $\sigma_w$ as $m$ grows. The shared bias $\beta$ stays invisible at every ensemble size. Coverage then plateaus at a level set by $|\beta|/\sigma_w$. Episode resampling, random seeds, and gradient diversification all enlarge $\sigma_w$ inside a single family. We claim they barely touch $\beta$ when the target policy visits regions the data does not cover.
Evaluators from different families share less of their bias: neural fitted Q-evaluation, fitted Q-evaluation with tree ensembles (which by Theisen et al. do not suffer the agreement collapse), model-based rollouts, and density-ratio estimators. With $F$ families, each contributing $m_f$ members, we form a random-effects interval whose variance is
$$\hat\sigma^2 = \hat\sigma_b^2 + \frac{1}{F}\sum_{f=1}^{F}\frac{\hat\sigma_{w,f}^2}{m_f}.$$
Here $\hat\sigma_b^2$ is the variance of the family means around the grand mean, and $\hat\sigma_{w,f}^2$ is the within-family spread of family $f$. The between-family term $\hat\sigma_b^2$ serves as an estimable stand-in for shared bias. The ratio $\rho = \hat\sigma_b^2/\bar\sigma_w^2$ can be computed without ground truth, where $\bar\sigma_w^2$ is the average within-family variance. It is our proposed early warning that homogeneous intervals will under-cover. The ratio plays the role for evaluation intervals that the disagreement-to-error ratio plays for classifier ensembles.
Research Program
The first step is a scaling study. On policies with known true values, we train neural fitted Q-evaluation ensembles of size 1 to 64, built three ways: random seeds only, episode-bootstrapped data, and seeds plus an adaptation of the gradient diversity penalty from Diversified Q-Ensemble to evaluation. For each target policy we record interval width, bias, and 90% interval coverage as functions of $m$. We fit $c + b m^{-a}$ to the width curve and test whether the fitted ceiling $c$, compared against held-out error, predicts the coverage plateau. The expected saturation form comes from the power-law results for classifier likelihood. Whether it holds for Bellman-trained regressors is itself an open question.
The second step builds the mixed-family evaluator at a matched compute budget, measured in total GPU-hours and parameters. We compare three aggregation rules on identical members: pooling all members into one spread, the random-effects rule above, and the convex hull of per-family intervals. This isolates the loss introduced by combination, which Morio et al. describe only for voting.
The third step tests whether $\rho$ is a useful diagnostic. Across target policies, does $\rho$, computed from logged data alone, rank-correlate with the coverage shortfall of the homogeneous ensemble? If it does, practitioners get a cheap signal for when adding more members of one family is wasted compute.
Baselines
The naive reference is a point estimate with no interval: neural fitted Q-evaluation from Benchmarks for Deep Off-Policy Evaluation, whose best methods reach rank correlation of roughly 0.69 on RL Unplugged and roughly 0.25 on heavily shifted D4RL. Any interval method whose lower bound ranks policies worse than this has failed on ranking.
The strongest same-family competitor is episode-bootstrapped fitted Q-evaluation from Hao et al. We run it with the same network, the same number of retrainings as our total member count, and the same learning-rate and training-length tuning budget, selected by held-out Bellman error. No coverage figure for it on DOPE-D4RL was found on Lacuna. Its known limit is under-coverage below K = 50 episodes. The gradient-diversified ensemble, adapting Diversified Q-Ensemble to evaluation, is the strongest attempt to push single-family diversity. It gets the same budget, with the penalty weight tuned over the same grid.
The strongest statistical baseline is the triply robust interval of Deeply-Debiased Off-Policy Interval Estimation. It keeps 0.9 nominal coverage even when its nuisance models converge as slowly as $(nT)^{-1/6}$, and we give it the same nuisance architectures. For policy selection, we also run the lower-bound ranking of Pessimistic Model Selection for Offline Deep Reinforcement Learning, which reports Pearson ρ = 0.895 against ρ = 0.609 for approximate-model point estimates on its own discrete-action benchmarks, with the same number of data chunks as our member count.
The key ablation uses the same mixed-family members pooled into one spread. It separates the gain from family diversity from the gain from the random-effects combination rule. We would count our method as winning if it reaches at least 0.9 empirical coverage on the high-shift D4RL policies where the homogeneous and bootstrap ensembles fall short. Its median width must stay within a stated factor of the triply robust interval, and the rank correlation of its lower bound must not fall below the point-estimate baseline.
Why not just…?
Why not just bootstrap episodes with a single architecture? Hao et al. prove it valid for linear function approximation, and this proposal's scaling study is designed to measure the deep, shifted case directly. If bootstrapped coverage rises to nominal as members are added, our shared-bias claim is wrong and we will report that. We expect a plateau, because resampling episodes changes which data each member sees but not how every member extrapolates beyond the data.
Why not just inflate the ensemble width by a constant calibrated on held-out data? This is the most serious threat. The behavior policy's own returns are observable in the logs, so one could calibrate a width multiplier there. We include this baseline. The experiment that settles it measures whether a multiplier calibrated on the behavior policy transfers to target policies at increasing distance from that policy. Our hypothesis says it will under-correct as shift grows, because shared bias is small in-distribution. If a single multiplier transfers well across DOPE's policy families, that simpler method wins.
Why not just use gradient diversification or a very large ensemble? Diversified Q-Ensemble shows the penalty matches a 500-member naive ensemble with 10 to 50 members in policy learning. The penalty spreads action-gradients within one function class, though, and leaves shared extrapolation bias in place. Beeson and Montana report 49 hours for a 500-member ensemble against 0.9 hours for 10, so brute force is expensive. Both appear on our coverage-versus-size curves at matched compute.
Why not just use the triply robust interval? It carries real guarantees and may win in low-shift settings. Its guarantees assume overlap between the logging and target policies, which Off-policy Evaluation Beyond Overlap shows can fail. The failure is exactly the far-shift regime we target. We report results separately by shift level.
Proposed Experiments
The main testbed is DOPE through google-research/deep_ope. It supplies target policies with ground-truth values on RL Unplugged (low shift) and D4RL Gym-MuJoCo, Adroit, and AntMaze (high shift, where all current methods fail at ranking). The OPCC benchmark tests paired comparisons, scored by the area under the risk-coverage curve. The Sepsis simulator from Hao et al. covers the regime with fewer than 50 episodes, and Mountain Car with linear features serves as a sanity check where bootstrap theory holds. Metrics are empirical coverage at 90%, median interval width, rank correlation and top-1 regret of lower bounds, and the correlation between $\rho$ and coverage shortfall.
Three results would be decisive. First, homogeneous coverage-versus-size curves should flatten below 0.9 on D4RL but not on RL Unplugged, with the plateau height tracking the fitted ceiling $c$. Second, at matched compute the mixed-family random-effects interval should reach nominal coverage where homogeneous ensembles do not, and the pooled-spread ablation should show how much of that gain comes from the combination rule. Third, $\rho$ should predict coverage shortfall across policies without access to true values. The core claim is refuted if any of the following holds: homogeneous coverage keeps climbing with ensemble size, the behavior-calibrated multiplier transfers across shift levels, or $\rho$ carries no signal.
References
- When are ensembles really effective?
- On Power Laws in Deep Ensembles
- Rethinking Fano's Inequality in Ensemble Learning
- Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
- Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning
- Offline Policy Comparison with Confidence: Benchmarks and Baselines
- Bootstrapping Fitted Q-Evaluation for Off-Policy Inference
- Pessimistic Model Selection for Offline Deep Reinforcement Learning
- On Local Posterior Structure in Deep Ensembles
- Benchmarks for Deep Off-Policy Evaluation
- Deeply-Debiased Off-Policy Interval Estimation
- Off-policy Evaluation Beyond Overlap: Sharp Partial Identification Under Smoothness