Demystifying Approximate RL with $\epsilon$-greedy Exploration: A Differential Inclusion View

Aditya Gopalan, Gugan ThoppeICLR 2023 · OpenReview

Abstract

Q-learning and SARSA(0) with $\epsilon$-greedy exploration are leading reinforcement learning methods, and their tabular forms converge to the optimal Q-function under reasonable conditions. However, with function approximation, they exhibit unexpected behaviors, such as i.) policy oscillation and chattering, and ii.) convergence to different attractors (possibly even the worst policy) on different runs, ii.) multiple attractors, and iii.) worst policy convergence, apart from the textbook instability. Accordingly, a theory to explain these phenomena has been a long-standing open problem, even for basic linear function approximation (Sutton, 1999). Our work uses differential inclusion theory to provide the first framework for resolving this problem. We further illustrate via numerical examples how this framework helps explain these algorithms' asymptotic behaviors.
BibTeX
@inproceedings{gopalan2023demystifying,
  author = {Aditya Gopalan and Gugan Thoppe},
  title = {{Demystifying Approximate RL with $\textbackslash\{\}epsilon$-greedy Exploration: A Differential Inclusion View}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=Ms1Zs8s7rg},
}

Loading related content...

Figure