Suppressing Overestimation in Q-Learning through Adversarial Behaviors

HyeAnn Lee, Donghwan LeeSubmission to ICLR 2024 · Withdrawn

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

The goal of this paper is to propose a new Q-learning algorithm with a dummy adversarial player, which is called dummy adversarial Q-learning (DAQ), that can effectively control the overestimation bias in standard Q-learning. With the dummy player, the learning can be formulated as a two-player zero-sum game. The proposed DAQ unifies several Q-learning variations to control overestimation, such as Maxmin Q-learning and Minmax Q-learning (proposed in this paper) in a single framework. The proposed DAQ is a simple but effective way to suppress the overestimation bias from subtle rewards and can be easily applied to off-the-shelf reinforcement learning algorithms. A finite-time convergence of DAQ is analyzed from an integrated perspective by adapting an adversarial Q-learning. The performance of the suggested DAQ is empirically demonstrated under various MDP environments.
BibTeX
@misc{lee2023suppressing,
  author = {HyeAnn Lee and Donghwan Lee},
  title = {{Suppressing Overestimation in Q-Learning through Adversarial Behaviors}},
  year = {2023},
  eprint = {2310.06286},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.06286},
}
Figure