An analysis of reinforcement learning with function approximation

Francisco S. Melo, Sean P. Meyn, M. Isabel Ribeiro

Abstract

We address the problem of computing the optimal Q-function in Markov decision problems with infinite state-space. We analyze the convergence properties of several variations of Q-learning when combined with function approximation, extending the analysis of TD-learning in Tsitsiklis and Van Roy (1996) to stochastic control settings. We identify conditions under which such approximate methods converge with probability 1. We conclude with a brief discussion on the general applicability of our results and compare them with several related works.
BibTeX
@inproceedings{melo2008analysis,
  author = {Francisco S. Melo and Sean P. Meyn and M. Isabel Ribeiro},
  title = {{An analysis of reinforcement learning with function approximation}},
  booktitle = {International Conference on Machine Learning},
  year = {2008},
  doi = {10.1145/1390156.1390240},
}

Loading related content...

Figure