Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
Lists
Only you see this
Abstract
We propose a new algorithm for model-based distributional reinforcement learning (RL), and prove that it is minimax-optimal for approximating return distributions in the generative model regime (up to logarithmic factors), the first result of this kind for any distributional RL algorithm. Our analysis also provides new theoretical perspectives on categorical approaches to distributional RL, as well as introducing a new distributional Bellman equation, the stochastic categorical CDF Bellman equation, which we expect to be of independent interest. Finally, we provide an experimental study comparing a variety of model-based distributional RL algorithms, with several key takeaways for practitioners.
BibTeX
@inproceedings{rowland2024near,
author = {Mark Rowland and Kevin Kevin Li and Rémi Munos and Clare Lyle and Yunhao Tang and Will Dabney},
title = {{Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model}},
booktitle = {Advances in Neural Information Processing Systems},
year = {2024},
url = {https://openreview.net/forum?id=JXKbf1d4ib},
}