Learning to Adapt in Dynamic, Real-World Environments Through\n Meta-Reinforcement Learning

Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, Chelsea Finn

Abstract

Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot. We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.

Introduction

Reinforcement Learning (RL) has demonstrated remarkable potential in controlled simulations, yet a significant "reality gap" persists. Physical robots face two primary hurdles: extreme sample inefficiency, where standard algorithms require millions of trials, and brittleness, where a policy trained for one specific environment fails when faced with hardware degradation or terrain changes. Rather than attempting to train a universal policy for every possible scenario—an intractable task—this research focuses on meta-learning: training an agent to learn how to adapt its behavior on the fly.

Methodology

The proposed framework combines model-based RL with meta-learning to enable rapid online adaptation. In model-based RL, an agent learns a dynamics model to predict future states based on current actions. This work innovates by treating short temporal windows of a trajectory as individual "tasks," allowing for adaptation within a single episode.

Two primary algorithms are introduced:

  • GrBAL (Gradient-Based Adaptive Learner): Utilizing a MAML-style approach, it meta-trains a model initialization such that a few gradient steps on recent history allow for accurate near-term predictions.
  • ReBAL (Recurrence-Based Adaptive Learner): This uses a Recurrent Neural Network (RNN) where the hidden state serves as the adaptation mechanism, adjusting predictions based on the immediate context.

At test time, the adapted model is paired with a Model Predictive Path Integral (MPPI) controller to plan optimal paths by simulating thousands of potential future trajectories.

Experimental Evaluation

The researchers evaluated the framework across diverse simulated and physical benchmarks. Simulation tasks included the Half-Cheetah (navigating slopes and floating blocks) and the Ant (managing crippled limbs). Real-world testing was conducted on a custom six-legged millirobot, which was trained on only 30 minutes of random movement across various surfaces.

The study utilizes both simulated MuJoCo environments (top/middle) and a real-world legged millirobot (bottom) to test adaptation to novel terrains and hardware failures.

Key Results and Discussion

The experimental results highlight a 1000x improvement in sample efficiency compared to state-of-the-art model-free meta-RL methods like MAML-RL. While model-free methods require days of experience, the proposed model-based approach achieves comparable performance in just a few hours.

Performance comparison on the Ant and Half-Cheetah tasks. The proposed GrBAL and ReBAL methods (green) reach high average returns significantly faster than model-free (TRPO, MAML-RL) or non-adaptive model-based baselines.

Furthermore, the adaptive model demonstrated superior robustness in dynamic environments. In the "floating pier" simulation, the adaptive model outperformed an "oracle" model trained with infinite data on that specific task, proving that local adaptation can compensate for stochastic changes better than a static global model. On the physical millirobot, the system successfully adapted to a missing leg, steep slopes, and varying payloads without any prior exposure to these specific conditions during training.

Conclusion

This work demonstrates that meta-learning a dynamics model for online adaptation is a viable path toward robust, real-world autonomous agents. While model-based methods may eventually plateau at lower asymptotic performance than model-free counterparts, their efficiency and ability to handle unpredictable perturbations make them ideal for physical robotics. Future work will focus on reducing hyperparameter sensitivity and exploring scenarios where environmental changes are more abrupt.

Summary updated

Abstract

Although reinforcement learning methods can achieve impressive results in\nsimulation, the real world presents two major challenges: generating samples is\nexceedingly expensive, and unexpected perturbations or unseen situations cause\nproficient but specialized policies to fail at test time. Given that it is\nimpractical to train separate policies to accommodate all situations the agent\nmay see in the real world, this work proposes to learn how to quickly and\neffectively adapt online to new tasks. To enable sample-efficient learning, we\nconsider learning online adaptation in the context of model-based reinforcement\nlearning. Our approach uses meta-learning to train a dynamics model prior such\nthat, when combined with recent data, this prior can be rapidly adapted to the\nlocal context. Our experiments demonstrate online adaptation for continuous\ncontrol tasks on both simulated and real-world agents. We first show simulated\nagents adapting their behavior online to novel terrains, crippled body parts,\nand highly-dynamic environments. We also illustrate the importance of\nincorporating online adaptation into autonomous agents that operate in the real\nworld by applying our method to a real dynamic legged millirobot. We\ndemonstrate the agent's learned ability to quickly adapt online to a missing\nleg, adjust to novel terrains and slopes, account for miscalibration or errors\nin pose estimation, and compensate for pulling payloads.\n
BibTeX
@inproceedings{nagabandi2019learning,
  author = {Anusha Nagabandi and Ignasi Clavera and Simin Liu and Ronald S. Fearing and Pieter Abbeel and Sergey Levine and Chelsea Finn},
  title = {{Learning to Adapt in Dynamic, Real-World Environments Through\textbackslash\{\}n Meta-Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=729wubut0o},
}
Figure