Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning

Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, Chelsea Finn

Introduction

Reinforcement Learning (RL) has achieved impressive results in simulated environments, but deploying these systems in the messy, unpredictable real world presents significant hurdles. Two major challenges remain: sample inefficiency and brittleness. Model-free RL often requires millions of trials, which is prohibitively expensive and slow for physical hardware. Furthermore, a policy trained to walk on flat ground often fails completely if the robot loses a limb, encounters a slope, or picks up a heavy payload.

Standard RL assumes the environment is stationary. While some "online" RL methods exist, they usually adapt too slowly to prevent a robot from failing when a sudden physical change occurs. To address this, researchers have introduced a model-based meta-reinforcement learning (meta-RL) framework that allows robots to adapt to unexpected physical changes—like a broken leg or a new terrain—in a matter of seconds. By treating short temporal segments as "tasks," this approach is orders of magnitude more sample-efficient than previous model-free meta-RL methods, enabling successful online adaptation on a real-world legged millirobot.

Methodology: Model-Based Meta-RL

Instead of learning a single "global" dynamics model that attempts to perfectly capture every possible environmental state, the authors propose meta-training a model specifically designed to be easy to adapt.

Redefining the "Task"

Unlike traditional meta-learning where a task might be defined as "running at a specific speed," this framework defines a task as a short temporal segment of experience. The model uses the past $M$ timesteps to adapt its parameters so it can accurately predict the next $K$ timesteps. This shifts adaptation from the episode level to the timestep level, allowing the agent to recover during a single run.

Two Flavors of Adaptation

The authors introduce two primary methods for online adaptation:

  • GrBAL (Gradient-Based Adaptive Learner): Built on the MAML (Model-Agnostic Meta-Learning) algorithm, GrBAL learns a set of initial weights such that taking just one or two gradient steps on recent data yields a highly accurate local dynamics model.
  • ReBAL (Recurrence-Based Adaptive Learner): This approach utilizes a Recurrent Neural Network (RNN) that "ingests" recent experience into its hidden state, automatically adjusting its predictions without explicit gradient updates at test time.

Control via Model Predictive Control (MPC)

Once the model is adapted to the current local context, the agent uses Model Predictive Path Integral (MPPI) control to plan its next moves. The agent essentially "imagines" various action sequences using the newly adapted local model and executes the optimal sequence.

Experimental Setup

The researchers evaluated their methods across both simulated and physical domains to test adaptability to sudden dynamic shifts.

Simulated and real-world environments used for evaluation.
  • Simulated Environments (MuJoCo): The framework was tested on a Half-Cheetah (facing disabled joints, sloped terrains, and a "pier" environment with floating blocks of varying buoyancy) and an Ant (facing crippled legs).
  • Real-World Hardware: The team deployed the system on a 6-legged millirobot (VelociRoACH). They collected approximately 30 minutes of data across three terrains (carpet, styrofoam, turf) to meta-train the model prior to testing.

Key Results

The integration of meta-learning with model-based RL yielded significant improvements in both sample efficiency and online adaptability.

Learning curves for Ant Crippled and Half-cheetah Disabled environments.
  • 1000x Efficiency: Because the model learns the underlying transition dynamics ($s, a \to s'$) rather than mapping states directly to optimal actions, it requires drastically less data. The proposed method achieved competitive performance with only 1.5–3 hours of real-world experience, whereas model-free meta-RL would require the equivalent of days or weeks.
  • Superior Adaptation: In simulations, GrBAL and ReBAL outperformed standard "Model-Based + Dynamic Evaluation" (which simply fine-tunes a standard model at test time). This proves that explicitly training a model for adaptability yields better recovery than standard fine-tuning.
  • Real-World Success: The physical millirobot successfully adapted online to several unmodeled disruptions. It learned to compensate for a missing leg, navigated novel slopes and terrains unseen during training, compensated for miscalibrated sensors, and adjusted to pulling unexpected payloads.

Limitations and Future Work

While highly effective for rapid adaptation, the approach has a few limitations:

  • Asymptotic Performance: Like many model-based methods, the peak performance eventually plateaus below the absolute maximum achievable by model-free methods if given infinite data.
  • Hyperparameter Sensitivity: The choice of $M$ (past context) and $K$ (prediction horizon) is critical. If $M$ is too small, the model lacks sufficient information to adapt; if it is too large, the "local" context may have already shifted.
  • Computational Overhead: Running gradient updates (GrBAL) or RNN passes (ReBAL) alongside an MPC planner at every timestep is computationally intensive, posing a challenge for small, low-power robots.

Future work will explore handling more drastic, non-continuous changes in dynamics and integrating uncertainty estimation directly into the meta-learning process to further robustify real-world robotic control.

Summary updated

Abstract

Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot: We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.
BibTeX
@inproceedings{nagabandi2019learning,
  author = {Anusha Nagabandi and Ignasi Clavera and Simin Liu and Ronald S. Fearing and Pieter Abbeel and Sergey Levine and Chelsea Finn},
  title = {{Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=729wubut0o},
}
Figure