Abstract
The recently proposed distribution correction estimation (DICE) family of estimators has advanced the state of the art in off-policy evaluation from behavior-agnostic data. While these estimators all perform some form of stationary distribution correction, they arise from different derivations and objective functions. In this paper, we unify these estimators as regularized Lagrangians of the same linear program. The unification allows us to expand the space of DICE estimators to new alternatives that demonstrate improved performance. More importantly, by analyzing the expanded space of estimators both mathematically and empirically we find that dual solutions offer greater flexibility in navigating the tradeoff between optimization stability and estimation bias, and generally provide superior estimates in practice.
Introduction
Off-Policy Evaluation (OPE) is a fundamental challenge in Reinforcement Learning, aiming to estimate the performance of a target policy using data generated by a different behavior policy. This is essential for safety-critical domains like healthcare or autonomous driving where live testing is prohibited. Recent "DICE" methods (e.g., DualDICE, GenDICE) address this by estimating density ratios to correct for distribution shifts. However, these methods were previously developed through disparate mathematical approaches. This work provides a "Grand Unified Theory," viewing DICE methods as regularized Lagrangians of a single linear program (LP).
The Unified Framework
The value of a policy can be expressed via a Primal LP (relating to state-action values, $Q$) or a Dual LP (relating to stationary distributions, $d$). By forming the Lagrangian and introducing regularization (such as $f$-divergences) and redundant constraints (e.g., normalization), the authors demonstrate that existing DICE methods are specific instances of this framework.
A key theoretical contribution is the analysis of bias. As shown in the table above, regularizing the primal variable ($Q$) often introduces permanent bias. In contrast, regularizing the dual variable ($\zeta$) provides optimization stability while remaining asymptotically unbiased, particularly when using the dual estimator $\hat{\rho}_\zeta$.
Empirical Results
The framework was evaluated on Gridworld, Cartpole, and Reacher benchmarks using both linear function approximation and neural networks. These tasks represent a range of complexity from simple discrete grids to continuous control.
The empirical results confirm the theoretical advantages of dual-variable approaches. As seen in the figures, the dual estimator with dual regularization (Dual est. / Dual reg.) consistently tracks the true value with lower variance and higher accuracy than primal or Lagrangian-based alternatives. This configuration, termed BestDICE, leverages redundant constraints to anchor the optimization process in stochastic environments, preventing the instability often seen in minimax optimization.
Key Insights
- Dual Robustness: Dual regularization is significantly more robust than primal regularization. While primal estimators struggle with reward scaling and shifts, dual estimators remain invariant to such transformations.
- Stability vs. Bias: Unconstrained primal forms, while easier to optimize, suffer from significant bias, especially in continuous action spaces like the Reacher task.
- Redundant Constraints: Constraints like "the distribution must sum to 1" are mathematically redundant but practically essential for stabilizing neural network training in the "wild" of stochastic gradient descent.
Conclusion
By unifying DICE methods under the regularized Lagrangian framework, this research clarifies the design space for OPE. The discovery of BestDICE and the demonstrated superiority of dual-variable regularization provide a more stable and accurate path for evaluating policies from offline data. Future work may explore automated hyperparameter tuning for regularization coefficients and relaxing ergodicity assumptions in complex, partitioned environments.
Summary updated