Using policy gradient reinforcement learning on autonomous robot controllers

Gregory Z. Grudic, Vijay Kumar, Lyle H. Ungar

TL;DR

This paper introduces a framework that uses Policy Gradient Reinforcement Learning (PGRL) to autonomously improve existing robot controllers. By treating controller parameters as learnable, the system can refine its behavior to achieve locally optimal performance, even in complex and uncertain environments, reducing task execution time and improving robustness.

What Problem Does This Paper Address?

Developing robust robot controllers that perform well across a wide range of conditions is a significant challenge. While programmers can often create controllers that work under specific, well-understood circumstances, achieving general robustness is difficult. Existing reinforcement learning (RL) approaches often struggle with the large, continuous state spaces and the infeasibility of millions of learning episodes common in real-world robotics. This paper addresses the need for a method that can take a pre-existing, potentially imperfect, controller and automatically tune its parameters to enhance its performance, shifting the burden of fine-tuning from human designers to an RL algorithm.

Methods

The core of the proposed framework is the application of Policy Gradient Reinforcement Learning (PGRL). Specifically, the authors utilize the Deterministic Policy Gradient (DPG) algorithm. The approach models the robot's learning problem as a Partially Observable Markov Decision Process (POMDP).

Here's a breakdown of the key methods:

  • Controller Representation: The robot's controller is represented by a set of real parameters, $\Theta = (\theta_1, ..., \theta_Q)$. Modifying these parameters changes the robot's behavior and, consequently, its performance.
  • Reward Function: A reward function, $p(\Theta)$, is defined to quantify performance. This function typically assigns positive rewards for achieving goals and negative rewards for undesirable events like collisions.
  • Policy Gradient Estimation: The DPG algorithm is used to estimate the gradient of the performance with respect to the controller parameters ($\partial \rho / \partial \Theta$). This gradient indicates the direction in parameter space that will most effectively increase the reward.
  • Parameter Update: The controller parameters are updated iteratively using gradient ascent: $\Theta^{t+1} = \Theta^t + \alpha \frac{\partial \rho}{\partial \Theta}$, where $\alpha$ is a learning rate. This process incrementally refines the controller towards a locally optimal policy.
  • Hybrid Control: The framework is applied to a hybrid control system, which consists of multiple discrete modes (e.g., "follow potential field," "avoid obstacle," "recovery from collision"). The PGRL approach is used to tune both the parameters within these modes and the parameters that govern the transitions between them.

What's new here is the direct application of PGRL to tune both the internal dynamics of controller modes and the switching logic between them, leveraging prior knowledge embedded in an initial controller specification.

Datasets

The paper does not explicitly mention using pre-existing datasets for training. Instead, the experiments are conducted using:

  • A simulated Cye robot environment: This simulator models the robot's geometry and kinematic constraints, including noise in position estimation.
  • The actual Cye robot: The learned policies are then tested on a physical Cye robot.

The "dataset" in this context is generated through the robot's interaction with its environment (both simulated and real) during the learning process.

Key Insights and Ideas

  • Bridging the Gap between Theory and Practice: The paper highlights that even theoretically robust controllers can perform poorly in practice due to unmodeled dynamics. RL offers a way to bridge this gap by fine-tuning controllers on the actual robot.
  • Leveraging Prior Knowledge: Instead of starting from scratch, the framework allows for the incorporation of existing control specifications, providing a strong starting point for learning and significantly reducing the learning burden.
  • Simultaneous Learning of Mode and Transition Parameters: A key insight is the ability of PGRL to simultaneously learn parameters that define behavior within modes (e.g., potential field parameters) and parameters that govern when to switch between modes (e.g., obstacle detection distances). This allows for holistic improvement of the hybrid controller.
  • Focus on Local Optimality: The authors acknowledge that finding a globally optimal policy in complex environments is often infeasible. Their approach focuses on achieving locally optimal performance, which is still highly beneficial for practical robot control.

Key Results

The experimental results on a Cye robot performing a room entry, navigation, and exit task demonstrate the effectiveness of the PGRL framework:

  • Reduced Task Execution Time: The learned controller reduced the overall time the robot spent in the room by approximately 14%.
  • Elimination of Collisions: The learning process completely eliminated obstacle collisions.
  • Improved Robustness: The learned policy compensated for unmodeled dynamics and changes in the environment, such as a moved obstacle and a newly appeared obstacle.
  • Convergence: The algorithm converged to a locally optimal policy in about 200 episodes in simulation and showed continued improvement on the real robot over 20 episodes.

Limitations and Future Work

  • Local Optima: The authors explicitly state that the algorithm is guaranteed to converge to locally optimal policies, not necessarily globally optimal ones. This is a deliberate trade-off for practicality in complex environments.
  • Computational Cost: While the paper doesn't dwell on it, PGRL can be computationally intensive, especially with a large number of parameters.
  • Future Work: The paper suggests that this framework can be extended to more complex tasks and environments. Further research could explore more sophisticated reward functions, more advanced RL algorithms, and methods for achieving better generalization across different tasks or environments.

Summary updated

Abstract

Robot programmers can often quickly program a robot to approximately execute a task under specific environment conditions. However, achieving robust performance under more general conditions is significantly more difficult. We propose a framework that starts with an existing control system and uses reinforcement feedback from the environment to autonomously improve the controller's performance. We use the policy gradient reinforcement learning (PGRL) framework, which estimates a gradient (in controller space) of improved reward, allowing the controller parameters to be incrementally updated to autonomously achieve locally optimal performance. Our approach is experimentally verified on a Cye robot executing a room entry and observation task, showing significant reduction in task execution time and robustness with respect to un-modelled changes in the environment.
BibTeX
@inproceedings{grudic2003using,
  author = {Gregory Z. Grudic and Vijay Kumar and Lyle H. Ungar},
  title = {{Using policy gradient reinforcement learning on autonomous robot controllers}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2003},
  doi = {10.1109/iros.2003.1250662},
}
Figure