Cooperative off-policy prediction of Markov decision processes in adaptive networks

Sergio Valcarcel Macua, Jianshu Chen, Santiago Zazo, Ali H. Sayed

Abstract

We apply diffusion strategies to propose a cooperative reinforcement learning algorithm, in which agents in a network communicate with their neighbors to improve predictions about their environment. The algorithm is suitable to learn off-policy even in large state spaces. We provide a mean-square-error performance analysis under constant step-sizes. The gain of cooperation in the form of more stability and less bias and variance in the prediction error, is illustrated in the context of a classical model. We show that the improvement in performance is especially significant when the behavior policy of the agents is different from the target policy under evaluation.
BibTeX
@inproceedings{macua2013cooperative,
  author = {Sergio Valcarcel Macua and Jianshu Chen and Santiago Zazo and Ali H. Sayed},
  title = {{Cooperative off-policy prediction of Markov decision processes in adaptive networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2013},
  doi = {10.1109/icassp.2013.6638519},
}

Loading related content...

Figure