Asynchronous Methods for Model-Based Reinforcement Learning

Yunzhi Zhang, Ignasi Clavera, Boren Tsai, Pieter Abbeel

Abstract

Significant progress has been made in the area of model-based reinforcement learning. State-of-the-art algorithms are now able to match the asymptotic performance of model-free methods while being significantly more data efficient. However, this success has come at a price: state-of-the-art model-based methods require significant computation interleaved with data collection, resulting in run times that take days, even if the amount of agent interaction might be just hours or even minutes. When considering the goal of learning in real-time on real robots, this means these state-of-the-art model-based algorithms still remain impractical. In this work, we propose an asynchronous framework for model-based reinforcement learning methods that brings down the run time of these algorithms to be just the data collection time. We evaluate our asynchronous framework on a range of standard MuJoCo benchmarks. We also evaluate our asynchronous framework on three real-world robotic manipulation tasks. We show how asynchronous learning not only speeds up learning w.r.t wall-clock time through parallelization, but also further reduces the sample complexity of model-based approaches by means of improving the exploration and by means of effectively avoiding the policy overfitting to the deficiencies of learned dynamics models.

Introduction

Reinforcement Learning (RL) has traditionally faced a significant trade-off between sample efficiency and computational overhead. Model-free methods, such as PPO or Soft Actor-Critic, are computationally efficient but require millions of environment interactions, making them unsuitable for physical hardware where wear and tear are concerns. Conversely, model-based reinforcement learning (MBRL) methods learn a dynamics model of the environment to "imagine" trajectories, drastically reducing the need for real-world samples.

Despite their sample efficiency, state-of-the-art MBRL methods suffer from poor "wall-clock" efficiency. Traditional implementations are strictly sequential: the agent collects data, stops to train the model, stops again to update the policy, and then resumes collection. This bottleneck means an agent requiring only two hours of physical experience might take two days of total training time. This work introduces an asynchronous framework that decouples these processes, allowing model-based methods to run in real-time.

The Asynchronous Framework

The core contribution is a fully asynchronous architecture where the three primary components of MBRL are separated into independent workers communicating via shared parameter servers.

The proposed asynchronous framework (left) decouples data collection, model learning, and policy improvement into independent workers, contrasting with the traditional sequential cycle (right).
  1. Data Collection Worker: This worker continuously interacts with the environment using the most recent policy parameters. It pushes the resulting trajectories into a shared data buffer.
  2. Model Learning Worker: This worker pulls data from the buffer to update an ensemble of neural network dynamics models, pushing updated parameters to a model server.
  3. Policy Improvement Worker: This worker pulls the latest model parameters to simulate trajectories and improve the policy (e.g., using TRPO or PPO), pushing updated policy parameters to a policy server.

By allowing these workers to run at their own pace, the framework eliminates idle time and enables the agent to learn as it acts.

Experimental Evaluation

The framework was tested on standard continuous control benchmarks and real-world robotic tasks:

  • MuJoCo Benchmarks: Ant, HalfCheetah, Hopper, and Walker2d.
  • Real-World Robotics: A physical PR2 robot performing reaching, shape matching, and Lego stacking tasks.

The researchers adapted three state-of-the-art algorithms—ME-TRPO, ME-PPO, and MB-MPO—into this asynchronous framework to demonstrate its versatility.

Results and Discussion

The asynchronous framework achieved massive speedups in wall-clock time. For instance, in the complex "Ant" locomotion task, the asynchronous version reached optimal performance in just 60 minutes, compared to over 10 hours for the sequential version.

Performance comparison across MuJoCo benchmarks. Asynchronous variants (solid lines) significantly outperform their synchronous counterparts (dashed/dotted lines) in terms of wall-clock time and often achieve higher final returns.

Beyond speed, the asynchronous approach yielded unexpected benefits in learning quality:

  • Natural Regularization: In synchronous MBRL, policies often overfit to the specific errors of a static model. In the asynchronous setting, the model is updated continuously while the policy is being improved. This dynamic environment acts as a regularizer, preventing the policy from exploiting model inaccuracies.
  • Enhanced Exploration: Since the data collection worker uses the latest policy for every rollout, the diversity of the collected data is higher than in batch-collected synchronous methods.
  • Sample Efficiency: Surprisingly, the asynchronous methods often required fewer total environment samples to reach peak performance. This suggests that the "messy" interaction between workers is fundamentally more efficient for learning.

Conclusion

This work demonstrates that asynchronous parallelization is a powerful tool for making model-based RL practical for real-world robotics. By reducing training time to match data collection time, the framework enables rapid iteration and real-time learning on physical hardware. Future research will focus on scaling these methods to higher-dimensional tasks and optimizing worker allocation to further enhance learning rates.

Summary updated

Abstract

Significant progress has been made in the area of model-based reinforcement learning. State-of-the-art algorithms are now able to match the asymptotic performance of model-free methods while being significantly more data efficient. However, this success has come at a price: state-of-the-art model-based methods require significant computation interleaved with data collection, resulting in run times that take days, even if the amount of agent interaction might be just hours or even minutes. When considering the goal of learning in real-time on real robots, this means these state-of-the-art model-based algorithms still remain impractical. In this work, we propose an asynchronous framework for model-based reinforcement learning methods that brings down the run time of these algorithms to be just the data collection time. We evaluate our asynchronous framework on a range of standard MuJoCo benchmarks. We also evaluate our asynchronous framework on three real-world robotic manipulation tasks. We show how asynchronous learning not only speeds up learning w.r.t wall-clock time through parallelization, but also further reduces the sample complexity of model-based approaches by means of improving the exploration and by means of effectively avoiding the policy overfitting to the deficiencies of learned dynamics models.
BibTeX
@inproceedings{zhang2019asynchronous,
  author = {Yunzhi Zhang and Ignasi Clavera and Boren Tsai and Pieter Abbeel},
  title = {{Asynchronous Methods for Model-Based Reinforcement Learning}},
  booktitle = {Conference on Robot Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=Vi_x3z7vP7F},
}
Figure