Learning to Adapt: Meta-Learning for Model-Based Control

Ignasi Clavera, Anusha Nagabandi, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, Chelsea Finn

Abstract

Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. This work proposes to learn how to quickly and effectively adapt online to new tasks using model-based meta-reinforcement learning. Our approach trains a dynamics model prior that can be rapidly adapted to the local context using recent data. Experiments demonstrate online adaptation for continuous control tasks on both simulated agents (adapting to novel terrains and crippled body parts) and a real-world dynamic legged millirobot (adapting to missing legs, novel terrains, and payloads).

Introduction

Reinforcement learning (RL) has achieved significant milestones in controlled simulations, yet its transition to real-world robotics is hindered by two primary obstacles: extreme sample inefficiency and brittleness to environmental changes. Standard RL algorithms typically assume a stationary environment and require millions of interactions to master a single task. In contrast, biological systems can adapt to physical changes—such as an injury or a change in terrain—almost instantaneously. This paper introduces a framework for Model-Based Meta-Reinforcement Learning that enables robots to use past experiences to learn the process of adaptation itself, allowing them to adjust to novel situations in real-time.

Methodology

The core of the proposed approach is to learn a dynamics model that is not necessarily globally accurate but is "easy to update." Instead of a static model, the authors use meta-learning to find a model prior that can be fine-tuned using a small window of recent experience.

Online Adaptation as Meta-Learning

The framework treats short temporal segments as individual "tasks." By using the last $M$ timesteps of data to update model parameters to predict the next $K$ timesteps, the agent can adapt to non-stationary conditions (e.g., moving from grass to concrete) as they occur. Two architectures were explored:

  • GrBAL (Gradient-Based Adaptive Learner): Utilizes gradient descent to update model parameters based on recent errors.
  • ReBAL (Recurrence-Based Adaptive Learner): Employs a Recurrent Neural Network (RNN) to maintain a hidden state that captures the current environmental context.

These adaptive models are integrated into a Model Predictive Control (MPC) loop, which performs real-time action planning based on the current, adapted understanding of the robot's dynamics.

Experimental Setup

The researchers evaluated their framework across a variety of simulated and physical platforms.

Simulated and real-world environments used for testing adaptation, including the Half-Cheetah, Ant, and VelociRoACH millirobot.

In simulation (MuJoCo), the "Half-Cheetah" and "Ant" robots were tested on challenging tasks such as navigating sloped terrains, crossing floating blocks with varying buoyancy, and operating with "crippled" or disabled joints. Real-world validation was conducted using the VelociRoACH, a six-legged millirobot known for its fast, stochastic dynamics. The robot was meta-trained on various surfaces and then tested on entirely unseen terrains and hardware failures.

Key Results and Discussion

The experimental results highlight the efficiency and resilience of the meta-RL approach.

Learning curves for the Ant Crippled and Half-cheetah Disabled tasks, showing that the proposed model-based meta-RL methods (GrBAL and ReBAL) achieve significantly higher returns with orders of magnitude fewer samples than model-free baselines.

Sample Efficiency and Performance

A standout finding is that the model-based meta-RL approach is approximately 1000x more sample efficient than model-free meta-RL counterparts. While model-free methods might require days of real-world data, this approach achieves high performance in just 1.5 to 3 hours of experience. Furthermore, in highly stochastic environments like the "floating pier," the adaptive model outperformed an "oracle" model trained specifically for that environment, as it could account for local, transient noise that a fixed model would treat as irreducible variance.

Real-World Resilience

On the physical millirobot, the system demonstrated remarkable robustness. When a leg was removed, the GrBAL-based controller successfully prevented the robot from drifting and allowed it to maintain its heading. It also enabled the robot to climb slopes steeper than those encountered during training and compensate for the added weight of pulling payloads.

Conclusion

This work demonstrates that training for adaptation rather than global accuracy is a powerful paradigm for real-world robotics. While model-based methods can sometimes plateau in asymptotic performance compared to model-free methods given infinite data, the gains in sample efficiency and online resilience are critical for autonomous agents operating in unpredictable environments. Future work will explore scaling these methods to higher-dimensional systems like humanoids and refining the selection of temporal context windows.

Summary updated

Abstract

Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot. We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.
BibTeX
@inproceedings{clavera2019learning,
  author = {Ignasi Clavera and Anusha Nagabandi and Ronald S. Fearing and Pieter Abbeel and Sergey Levine and Chelsea Finn},
  title = {{Learning to Adapt: Meta-Learning for Model-Based Control}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=729wubut0o},
}
Figure