Distributional off-policy evaluation with Bellman residual minimization

Sungee Hong, Zhengling Qi, Raymond K. W. WongSubmission to ICLR 2024 · Rejected

Abstract

We consider the problem of distributional off-policy evaluation which serves as the foundation of many distributional reinforcement learning (DRL) algorithms. In contrast to most existing works (that rely on supremum-extended statistical distances), we study the expectation-extended statistical distance for quantifying the Bellman residuals and provide the corresponding theoretical supports. Extending the framework of Bellman residual minimization to DRL, we propose a method called Energy Bellman Residual Minimization (EBRM) to estimate the return distribution. We establish a finite-sample error bound for the EBRM estimator under a realizability assumption. Additionally, we introduce a variant of our method based on a multi-step bootstrapping procedure to enable multi-step extension. By selecting an appropriate step level, we obtain a better error bound for this variant of EBRM compared to a single-step EBRM, under non-realizability settings. Finally, we demonstrate the superior performance of our method through simulation studies, comparing it to other existing methods.
BibTeX
@inproceedings{hong2025distributional,
  author = {Sungee Hong and Zhengling Qi and Raymond K. W. Wong},
  title = {{Distributional off-policy evaluation with Bellman residual minimization}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=U57CpgcIeL},
}
Figure