Energy-based Predictive Representation for Reinforcement Learning
Lists
Only you see this
Abstract
In real world applications, it is usually necessary for a reinforcement learning algorithm to handle the partial observability beyond Markov decision processes (MDPs). Although the partially observable Markov decision process (POMDP) has been precisely motivated for this requirement, such a formulation raises significant computational and statistical hardness challenges in learning and planning. In this work, we introduce the Energy-based Predictive Representation (EPR), which leads to a unified framework for practical reinforcement learning algorithm design in both MDPs and POMDPs settings, to handle the learning, exploration, and planning in a coherent way. The proposed approach relies on the powerful neural energy-based model to extract sufficient representation, from which Q-functions can be efficiently approximated. With such a representation, we develop an efficient approach for computing confidence, which allows optimism/pessimism in the face of uncertainty to be efficiently implemented in planning, hence managing the exploration versus exploitation tradeoff. An experimental investigation shows that the proposed algorithm can surpass state-of-the-art performance in both MDP and POMDP settings in comparison to existing baselines.
BibTeX
@inproceedings{zhang2023energy,
author = {Tianjun Zhang and Tongzheng Ren and Chenjun Xiao and Wenli Xiao and Joseph E. Gonzalez and Dale Schuurmans and Bo Dai},
title = {{Energy-based Predictive Representation for Reinforcement Learning}},
booktitle = {International Conference on Learning Representations},
year = {2023},
url = {https://openreview.net/forum?id=aCCRmE3Pglv},
}Loading related content...