Abstract
Significant progress has been made in the area of model-based reinforcement learning (MBRL). State-of-the-art algorithms are now able to match the asymptotic performance of model-free methods while being significantly more sample-efficient. However, the sequential nature of traditional MBRL—alternating between data collection, model training, and policy optimization—creates a substantial wall-clock time bottleneck. This paper introduces an asynchronous framework for MBRL that parallelizes these phases, reducing training time from days to hours. We demonstrate that this asynchrony not only speeds up learning but also acts as a regularizer, improving sample efficiency and robustness against model inaccuracies.
Introduction
Model-based reinforcement learning is often preferred for its data efficiency. By learning a dynamics model of the environment, agents can "imagine" trajectories and optimize policies without constant physical interaction. Despite this advantage, sequential MBRL implementations are computationally expensive. A robot might require only a few hours of interaction to learn a task, yet the total wall-clock time can exceed 48 hours due to the overhead of training complex neural network models and optimizing policies between collection episodes. This lag hinders real-time learning and rapid iteration for researchers.
Methodology: The Asynchronous Framework
To address the computational bottleneck, we propose a framework that decouples the three primary components of MBRL into independent, parallel workers. These workers communicate asynchronously through shared data and parameter servers.
The framework consists of three core components:
- Data Collection Worker: This worker continuously executes the current policy in the environment to collect trajectories, which are then pushed to a shared data buffer.
- Model Learning Worker: It pulls data from the buffer to train an ensemble of dynamics models. Training occurs one epoch at a time, with updated weights pushed to a model server. It utilizes early-stopping to prevent overfitting during early stages of data collection.
- Policy Improvement Worker: This worker pulls the latest model parameters to generate "imagined" samples and performs gradient steps to update the policy, pushing new parameters back to the policy server.
Unlike previous asynchronous methods like A3C, which parallelize data collection across multiple agents, our approach parallelizes the internal phases of the MBRL algorithm itself. This is particularly beneficial for robotics, where hardware resources are often limited to a single agent.
Experimental Evaluation
The framework was evaluated on standard MuJoCo simulation benchmarks (Ant, HalfCheetah, Hopper, and Walker2d) and a real-world PR2 robot. The PR2 tasks included contact-rich manipulation such as reaching, shape matching, and Lego stacking. These experiments aimed to measure both wall-clock time speedup and sample efficiency compared to sequential baselines like ME-TRPO and MB-MPO.
Results and Analysis
The asynchronous framework achieved a dramatic reduction in wall-clock time. For instance, in the complex Ant environment, the optimal policy was learned in approximately 60 minutes, a 10x speedup over the 10+ hours required by sequential methods. On the PR2 robot, tasks like Lego stacking were mastered in just 10 minutes of wall-clock time.
Beyond speed, the asynchronous approach demonstrated superior sample complexity. We identify several key factors contributing to this performance:
- Asynchrony as Regularization: In sequential MBRL, policies often overfit to the errors of a static, fully-trained model. In our framework, the policy is updated against a continuously evolving model. This "moving target" prevents the policy from committing to model biases, leading to more robust learning.
- Enhanced Exploration: Continuous policy updates mean the data collection worker uses a slightly different policy for each rollout, naturally diversifying the collected data and improving the global accuracy of the dynamics model.
- The Sampling Paradox: We observed that slower data collection can sometimes improve results by allowing the model and policy workers more gradient steps per environment sample, effectively giving the system more "thinking time" per unit of experience.
Conclusion
By parallelizing the core components of model-based reinforcement learning, we have developed a framework that makes wall-clock training time nearly equivalent to physical data collection time. This advancement enables real-time learning on physical robots and significantly reduces the barrier to hyperparameter tuning. Future work will focus on automated load balancing between workers to further stabilize learning in increasingly complex, long-horizon tasks.
Summary updated