Deep predictive policy training using reinforcement learning

Ali Ghadirzadeh, Atsuto Maki, Danica Kragic, Mårten Björkman

Abstract

Researchers at KTH Royal Institute of Technology have developed Deep Predictive Policy Training (DPPT), a hierarchical framework designed to overcome the data-hungry nature of standard Deep Reinforcement Learning (RL) in robotics. By decoupling vision, motor behavior, and task logic into three specialized neural layers, the system enables a PR2 robot to learn complex, high-speed tasks such as ball throwing and object grasping in approximately one hour of real-world training. Using only 180 physical attempts and simple qualitative rewards, the DPPT framework demonstrates a significant leap in data efficiency and practical robot autonomy.

Introduction

Most contemporary robotic systems operate on a reactive loop: they continuously process sensor input to calculate and execute immediate responses. However, sensor delays and processing latency often result in slow, jerky movements that contrast sharply with fluid human motion. Humans are inherently predictive; for a task like throwing a ball, we capture a visual snapshot of the target and execute a pre-planned, fluid motor trajectory rather than adjusting our arm position millisecond-by-millisecond based on visual feedback.

While training predictive policies using Deep RL is a promising path toward human-like robot performance, it is traditionally hindered by extreme data requirements and the difficulty of engineering precise reward functions. This paper introduces a method to make Deep RL data-efficient enough for physical hardware while utilizing human-friendly, qualitative rewards.

The DPPT Framework

The core of the proposed method is a hierarchical architecture consisting of three "super-layers" that separate the problems of perception, motion, and decision-making:

  1. Perception Super-layer: A Convolutional Spatial Autoencoder compresses raw RGB images into a low-dimensional state representing the 2D coordinates of relevant objects. This layer is pre-trained on synthetic images to ensure it can ignore background distractors.
  2. Behavior Super-layer: A Variational Autoencoder (VAE) represents complex, multi-joint motor trajectories as points within a 5D "action manifold." This layer is pre-trained in a physics simulator (Gazebo) to learn the underlying structure of valid, smooth motions.
  3. Policy Super-layer: This small "brain" layer maps the compressed visual state to a specific motor trajectory on the action manifold. It is trained on the physical robot using Trust Region Policy Optimization (TRPO).

By pre-training the vision and motor layers independently, the robot only needs to learn the specific mapping between them during the expensive real-world training phase.

Experimental Setup and Datasets

The researchers utilized a combination of synthetic, simulated, and real-world data to train the PR2 robot:

  • Synthetic Vision: 5,400 generated images of a target object (a Pikachu toy) were used to train the Perception layer to handle various positions and distractors.
  • Simulated Motor Data: 10,000 trajectory samples were gathered in the Gazebo simulator to define the Behavior layer's action manifold for throwing and grasping.
  • Real-world Trials: The final policy was refined through approximately 180 physical attempts per task on a PR2 robot.

Key Results

The DPPT framework achieved remarkable efficiency, with the robot learning to throw and grasp in roughly one hour of data collection. In simulation, TRPO proved to be the most stable and efficient algorithm for this architecture compared to alternatives like REPS or VPG.

Sequence of a PR2 robot performing a ball-throwing task using the DPPT framework.

In the ball-throwing task, the robot achieved an average reward of 1.58 (out of a maximum 2.0), indicating consistent success in hitting the target. A critical insight from the study is that qualitative rewards (e.g., human labels like "good" or "excellent") are sufficient for training, making the system accessible to non-expert users. Furthermore, the use of manifolds allowed for successful sim-to-real transfer, as the underlying structure of motion learned in simulation remained valid in the physical environment.

Discussion and Future Work

While the system is highly efficient, its primary limitation lies in perceptual robustness; entirely novel objects not seen during pre-training can occasionally confuse the spatial autoencoder. Future research will focus on expanding the variety of pre-training imagery to improve robustness. Additionally, the researchers aim to replace the initial "blind controllers" used in simulation with human motion capture or reactive controllers to further increase the system's autonomy from the outset.

Summary updated

BibTeX
@inproceedings{ghadirzadeh2017deep,
  author = {Ali Ghadirzadeh and Atsuto Maki and Danica Kragic and Mårten Björkman},
  title = {{Deep predictive policy training using reinforcement learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2017},
  doi = {10.1109/iros.2017.8206046},
}
Figure