Robo-PlaNet: Learning to Poke in a Day

Maxime Chevalier-Boisvert, Guillaume Alain, Florian Golemo, Derek NowrouzezahraiarXiv 2019 · OpenReview

TL;DR

This paper introduces Robo-PlaNet, an asynchronous version of the PlaNet model-based reinforcement learning approach. By parallelizing data collection and model training, Robo-PlaNet significantly reduces training time, enabling robots to learn tasks much faster than the original PlaNet. This makes model-based RL more practical for real-world robotic applications.

What Problem Does This Paper Address?

Learning complex robotic tasks directly from raw pixel observations is challenging. Model-based reinforcement learning methods like PlaNet offer a promising solution by learning a predictive model of the environment's dynamics. However, the original PlaNet architecture requires alternating phases of data collection (robot rollouts) and model training. This sequential process leads to significant robot downtime, making training impractically long, especially for real-world robots. The authors aim to overcome this bottleneck by making the training process asynchronous, thereby reducing robot idle time and accelerating learning.

Methods

The core innovation of Robo-PlaNet is its asynchronous architecture, which breaks down the original PlaNet into three independent, parallel processes:

  1. Roller Process (GPU): This process is responsible for interacting with the environment, collecting robot rollouts using the latest learned model. It receives updated model parameters and sends collected data to the Replay Buffer Process.
  2. Replay Buffer Process (CPU): This process manages the storage of collected trajectories. It receives data from the Roller Process, samples minibatches, and sends them to the Learner Process.
  3. Learner Process (GPU): This process trains the core PlaNet networks: the Variational Autoencoder (VAE), the reward estimator, and the recurrent neural network (RNN) transition model. It receives minibatches from the Replay Buffer Process and pushes updated model parameters to the Roller Process.

This parallelization ensures that the robot (Roller Process) can continuously collect data while the models are being trained (Learner Process), minimizing downtime. The authors adapted the original PlaNet's VAE to handle concatenated image inputs from two cameras, a necessary modification for their experimental setup.

Datasets

The experiments were conducted in two environments:

  • Simulation: A simulated Poppy Ergo Jr. robot environment using PyBullet.
  • Real Robot: A physical 6-Do-of-Freedom (DoF) Poppy Ergo Jr. robot.

The task in both environments was a reaching task where the robot had to touch a randomly placed cube. The authors note that they did not transfer policies from simulation to the real robot, but rather evaluated the method independently in each setting.

Key Insights and Ideas

  • Asynchronous Training is Key: The most significant insight is that by decoupling data collection and model training, the robot's downtime can be drastically reduced, leading to faster learning. This addresses a major practical limitation of previous model-based RL approaches on physical systems.
  • Real-World Robot Dynamics are Tricky: The authors highlight that real-world robots, especially low-cost ones like the Poppy Ergo Jr., exhibit complex dynamics (e.g., backlash) that are difficult to perfectly model. This underscores the need for learning directly from real robot data.
  • Camera Fusion for Better Observation: Using two cameras and fusing their observations late in the network proved effective for the reaching task, preventing the robot from obstructing its own view of the target. This approach is noted as being extensible to more cameras.
  • Hyperparameter Sensitivity: While Robo-PlaNet generally outperforms vanilla PlaNet, the authors observed that the original PlaNet's performance could be sensitive to its "collect interval" hyperparameter, and that one specific interval setting performed best on the real robot, possibly due to overfitting to non-stationary dynamics.

Key Results

Robo-PlaNet consistently achieved higher performance in the same amount of training time compared to the vanilla PlaNet implementation.

  • Simulation: Robo-PlaNet reached the point of solving the environment (poking the cube) significantly faster than PlaNet, even when PlaNet was configured with different "collect interval" hyperparameters (CI05, CI10, CI20).
  • Real Robot: Similar to simulation, Robo-PlaNet demonstrated a noticeable reduction in training time until convergence on the physical robot. The plots show that Robo-PlaNet reached a higher reward level within the same wall-clock time.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Task and Robot Specificity: The current work demonstrates performance on a single task and a single robot platform. Future work should extend Robo-PlaNet to a wider variety of robotic tasks and hardware.
  • Dense Reward Dependency: Like PlaNet, Robo-PlaNet relies on the environment providing a dense reward signal. This severely limits its applicability to tasks where such rewards are not readily available. The authors believe developing better methods for learning reward signals, especially for sparse reward settings, is crucial for enabling robots to learn from scratch.
  • Human Demonstrations: The authors suggest that incorporating human demonstrations could further accelerate learning, especially given the need for human presence during much of the robot's exploration.
  • Multi-Robot Systems: They also propose exploring multi-robot architectures, similar to GORILA, to further speed up training.

Summary updated

Abstract

Recently, the Deep Planning Network (PlaNet) approach was introduced as a model-based reinforcement learning method that learns environment dynamics directly from pixel observations. This architecture is useful for learning tasks in which either the agent does not have access to meaningful states (like position/velocity of robotic joints) or where the observed states significantly deviate from the physical state of the agent (which is commonly the case in low-cost robots in the form of backlash or noisy joint readings). PlaNet, by design, interleaves phases of training the dynamics model with phases of collecting more data on the target environment, leading to long training times. In this work, we introduce Robo-PlaNet, an asynchronous version of PlaNet. This algorithm consistently reaches higher performance in the same amount of time, which we demonstrate in both a simulated and a real robotic experiment.
BibTeX
@misc{chevalierboisvert2019robo,
  author = {Maxime Chevalier-Boisvert and Guillaume Alain and Florian Golemo and Derek Nowrouzezahrai},
  title = {{Robo-PlaNet: Learning to Poke in a Day}},
  year = {2019},
  eprint = {1911.03594},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.03594},
}
Figure