Learning Diverse and Effective Policies with Non-Markovian Rewards
Lists
Only you see this
Abstract
Learning a set of diverse and high-quantity policies is a difficult problem in Reinforcement Learning since the diversity of policies is demanded to be achieved without dampening their effectiveness. This problem becomes more challenging when the rewards are non-Markovian, i.e., the rewards depend on the history of states and actions, which are quite sparse and returned over a long period. The sparse supervision signals and the non-Markovian properties of the rewards hinder the learning of policy embeddings and thus the learning of diverse and high-quality policies. In this paper, we propose to use a diversity matrix to quantify policy diversity and theoretically prove that if the diversity matrix is positive definite, then the diversity of policies can be achieved without sacrificing their effectiveness. The policy diversity matrix stems from policy embeddings. To obtain high-quality embeddings, we adopt a transformer to capture mutual dependencies between states and actions and design pseudo tasks to overcome sparse rewards. Experimental results show that our method can achieve a set of policies with more effective diversity and better performance than multiple recently proposed baseline methods in a variety of non-Markovian and Markovian environments.
BibTeX
@misc{qu2022learning,
author = {Bohao Qu and Jielong Yang and Xiaoqiang Ren and Hechang Chen and Jifeng Hu and Xiaofeng Cao and SHENG ZHANG and Yi Chang},
title = {{Learning Diverse and Effective Policies with Non-Markovian Rewards}},
year = {2022},
url = {https://openreview.net/forum?id=w7ds3UKtQJl},
howpublished = {OpenReview},
}Loading related content...