Following Newton direction in Policy Gradient with parameter exploration
Lists
Only you see this
Abstract
This paper investigates the use of second-order methods to solve Markov Decision Processes (MDPs). Despite the popularity of second-order methods in optimization literature, so far little attention has been paid to the extension of such techniques to face sequential decision problems. Here we provide a model-free Reinforcement Learning method that estimates the Newton direction by sampling directly in the parameter space. In order to compute the Newton direction we provide the formulation of the Hessian of the expected return, a technique for variance reduction in the sample-based estimation and a finite sample analysis in the case of Normal distribution. Beside discussing the theoretical properties, we empirically evaluate the method on an instructional linear-quadratic regulator and on a complex dynamical quadrotor system.
BibTeX
@inproceedings{manganini2015following,
author = {Giorgio Manganini and Matteo Pirotta and Marcello Restelli and Luca Bascetta},
title = {{Following Newton direction in Policy Gradient with parameter exploration}},
booktitle = {IJCNN},
year = {2015},
doi = {10.1109/ijcnn.2015.7280673},
}Loading related content...