Abstract
Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. This work proposes to learn how to quickly and effectively adapt online to new tasks using model-based meta-reinforcement learning. Our approach trains a dynamics model prior that can be rapidly adapted to the local context using recent data. Experiments demonstrate online adaptation for continuous control tasks on both simulated agents (adapting to novel terrains and crippled body parts) and a real-world dynamic legged millirobot (adapting to missing legs, novel terrains, and payloads).
Introduction
Reinforcement learning (RL) has achieved significant milestones in controlled simulations, yet its transition to real-world robotics is hindered by two primary obstacles: extreme sample inefficiency and brittleness to environmental changes. Standard RL algorithms typically assume a stationary environment and require millions of interactions to master a single task. In contrast, biological systems can adapt to physical changes—such as an injury or a change in terrain—almost instantaneously. This paper introduces a framework for Model-Based Meta-Reinforcement Learning that enables robots to use past experiences to learn the process of adaptation itself, allowing them to adjust to novel situations in real-time.
Methodology
The core of the proposed approach is to learn a dynamics model that is not necessarily globally accurate but is "easy to update." Instead of a static model, the authors use meta-learning to find a model prior that can be fine-tuned using a small window of recent experience.
Online Adaptation as Meta-Learning
The framework treats short temporal segments as individual "tasks." By using the last $M$ timesteps of data to update model parameters to predict the next $K$ timesteps, the agent can adapt to non-stationary conditions (e.g., moving from grass to concrete) as they occur. Two architectures were explored:
- GrBAL (Gradient-Based Adaptive Learner): Utilizes gradient descent to update model parameters based on recent errors.
- ReBAL (Recurrence-Based Adaptive Learner): Employs a Recurrent Neural Network (RNN) to maintain a hidden state that captures the current environmental context.
These adaptive models are integrated into a Model Predictive Control (MPC) loop, which performs real-time action planning based on the current, adapted understanding of the robot's dynamics.
Experimental Setup
The researchers evaluated their framework across a variety of simulated and physical platforms.
In simulation (MuJoCo), the "Half-Cheetah" and "Ant" robots were tested on challenging tasks such as navigating sloped terrains, crossing floating blocks with varying buoyancy, and operating with "crippled" or disabled joints. Real-world validation was conducted using the VelociRoACH, a six-legged millirobot known for its fast, stochastic dynamics. The robot was meta-trained on various surfaces and then tested on entirely unseen terrains and hardware failures.
Key Results and Discussion
The experimental results highlight the efficiency and resilience of the meta-RL approach.
Sample Efficiency and Performance
A standout finding is that the model-based meta-RL approach is approximately 1000x more sample efficient than model-free meta-RL counterparts. While model-free methods might require days of real-world data, this approach achieves high performance in just 1.5 to 3 hours of experience. Furthermore, in highly stochastic environments like the "floating pier," the adaptive model outperformed an "oracle" model trained specifically for that environment, as it could account for local, transient noise that a fixed model would treat as irreducible variance.
Real-World Resilience
On the physical millirobot, the system demonstrated remarkable robustness. When a leg was removed, the GrBAL-based controller successfully prevented the robot from drifting and allowed it to maintain its heading. It also enabled the robot to climb slopes steeper than those encountered during training and compensate for the added weight of pulling payloads.
Conclusion
This work demonstrates that training for adaptation rather than global accuracy is a powerful paradigm for real-world robotics. While model-based methods can sometimes plateau in asymptotic performance compared to model-free methods given infinite data, the gains in sample efficiency and online resilience are critical for autonomous agents operating in unpredictable environments. Future work will explore scaling these methods to higher-dimensional systems like humanoids and refining the selection of temporal context windows.
Summary updated