Abstract
We consider the problem of distributional off-policy evaluation which serves as the foundation of many distributional reinforcement learning (RL) algorithms. In contrast to most existing works that rely on supremum-extended statistical distances, we study the expectation-extended statistical distance for quantifying the distributional Bellman residuals and show that it can upper bound the expected error of estimating the return distribution. Based on this property, we propose Energy Bellman Residual Minimizer (EBRM). We establish a finite-sample error bound under the realizability assumption and introduce a multi-step variant to handle non-realizability. Simulation studies demonstrate superior performance compared to existing methods.
1. Introduction
Off-Policy Evaluation (OPE) is a critical task in reinforcement learning, enabling practitioners to estimate the performance of a target policy using historical data collected by a different behavior policy. While traditional OPE focuses on the expected value of returns, Distributional RL aims to characterize the entire return distribution. This is essential for risk-sensitive applications such as autonomous driving or financial trading, where understanding the variance and tail behavior of returns is as important as the mean.
However, existing distributional OPE methods face significant challenges. Computationally, many rely on "supremum-extended" distances (e.g., supremum-Wasserstein), which require finding the maximum error across all states—a task that becomes intractable in continuous or high-dimensional spaces. Theoretically, most methods depend on the "completeness" assumption, which requires the model space to be closed under the Bellman operator. In practice, models are often misspecified, and increasing model expressivity can paradoxically break completeness.
2. The EBRM Framework
We introduce Energy Bellman Residual Minimization (EBRM) to address these limitations. EBRM shifts the focus from "worst-case" errors to "average-case" errors by utilizing expectation-extended distances.
2.1 Energy Distance and Bellman Residuals
A core component of our method is the use of the Energy Distance, a statistical metric related to Maximum Mean Discrepancy (MMD). The Energy Distance is particularly advantageous because it can be computed efficiently for multi-dimensional rewards and does not require the return distributions to have a closed-form density. By minimizing the difference (the "residual") between the two sides of the Distributional Bellman Equation under this metric, we can effectively recover the true return distribution.
2.2 Realizability over Completeness
A major theoretical contribution of this work is the relaxation of the "completeness" assumption. We prove that EBRM functions effectively under the "realizability" assumption—the condition that the true distribution is contained within the chosen model space. Realizability is a significantly weaker and more realistic requirement than completeness.
As illustrated in the table above, EBRM is the only method among those compared that provides a statistical error bound while simultaneously supporting multi-dimensional rewards and operating without the need for the completeness assumption.
3. Multi-step Extension for Non-Realizability
In scenarios where the model is misspecified (non-realizability), a single-step Bellman residual might lead to biased estimates. To mitigate this, we propose a multi-step version of EBRM. By looking $m$ steps into the future, the algorithm can "bypass" some of the errors introduced by the model's limitations. We provide a practical bootstrap-based implementation of this multi-step approach, which acts as a bridge to pull the estimate closer to the best possible approximation within the model space.
4. Experimental Results
We evaluated EBRM using standard benchmarks and synthetic environments designed to test theoretical limits.
4.1 Benchmark Performance
In OpenAI Gym tasks (Acrobot, Cartpole, and MountainCar), EBRM consistently achieved lower inaccuracy compared to established baselines like QRDQN and MMDRL. Accuracy was measured using both Energy and Wasserstein distances, confirming the robustness of our approach across different metrics.
4.2 Robustness to Misspecification
Using a synthetic 30-state environment specifically designed to violate the completeness assumption, we compared EBRM against Fitted Likelihood Estimation (FLE). While FLE's performance degraded as more data was added—converging to an incorrect distribution due to the lack of completeness—EBRM successfully converged to the best possible distribution within the model space.
4.3 Statistical Guarantees
Our theoretical analysis provides rigorous finite-sample error bounds. We show that the error of the EBRM estimator shrinks at a rate of approximately $1/\sqrt{N}$, where $N$ is the number of samples. This provides a reliable statistical guarantee for practitioners deploying these models in real-world scenarios.
5. Conclusion
EBRM offers a robust and computationally efficient framework for distributional OPE. By leveraging expectation-extended distances and relaxing the completeness assumption, it overcomes the primary hurdles faced by previous methods. While the multi-step variant requires trajectory data and introduces a bias-variance trade-off in selecting the step-size $m$, EBRM represents a significant step forward in making distributional RL more applicable to complex, real-world problems.
Summary updated