Wide-Sense Stationary Policy Optimization with Bellman Residual on Video Games
Lists
Only you see this
Abstract
Deep Reinforcement Learning (DRL) has an increasing application in video games. However, it usually suffers from unstable training, low sampling efficiency, etc. Under the assumption that Bellman residual follows a stationary random process when the training process is convergent, we propose the Wide-sense Stationary Policy Optimization (WSPO) framework, which leverages the Wasserstein distance from the Bellman Residual Distribution (BRD) between two adjacent time steps, to stabilize the training stage and improve the sampling efficiency. We minimize the Wasserstein distance with Quantile Regression, where the specific form of BRD is not needed. Finally, we combine WSPO with Advantage Actor-Critic (A2C) algorithm and Deep Deterministic Policy Gradient (DDPG) algorithm. We evaluate WSPO on Atari 2600 video games and continuous control tasks, illustrating that WSPO compares or outperforms the state-of-the-art algorithms we tested.
BibTeX
@inproceedings{gong2021wide,
author = {Chen Gong and Qiang He and Yunpeng Bai and Xinwen Hou and Guoliang Fan and Yu Liu},
title = {{Wide-Sense Stationary Policy Optimization with Bellman Residual on Video Games}},
booktitle = {ICME},
year = {2021},
url = {https://openreview.net/forum?id=APhjix1itO},
}Loading related content...