Meta Reinforcement Learning with Latent Variable Gaussian Processes

Steindór Sæmundsson, Katja Hofmann, Marc Peter Deisenroth

Abstract

Learning from small data sets is critical in many practical applications where data collection is time consuming or expensive, e.g., robotics, animal experiments or drug design. Meta learning is one way to increase the data efficiency of learning algorithms by generalizing learned concepts from a set of training tasks to unseen, but related, tasks. Often, this relationship between tasks is hard coded or relies in some other way on human expertise. In this paper, we frame meta learning as a hierarchical latent variable model and infer the relationship between tasks automatically from data. We apply our framework in a model-based reinforcement learning setting and show that our meta-learning model effectively generalizes to novel tasks by identifying how new tasks relate to prior ones from minimal data. This results in up to a 60% reduction in the average interaction time needed to solve tasks compared to strong baselines.

Introduction

Reinforcement learning (RL) has achieved remarkable success in simulated environments, yet its application to real-world robotics remains limited by extreme data requirements. Traditional RL agents often require millions of interactions to master simple tasks, a process that is both time-consuming and taxing on physical hardware. Meta-learning addresses this by enabling agents to "learn how to learn," leveraging experience from a distribution of related tasks to accelerate learning in new environments.

However, many existing meta-RL methods rely on human-defined task relationships or require vast amounts of data to infer these connections. This paper introduces Meta-Learning Gaussian Processes (ML-GP), a framework designed for data-efficient meta-RL. The core idea is to disentangle the universal laws of physics from task-specific parameters—such as the mass of a robot arm or the friction of a joint—allowing for rapid adaptation to novel tasks with minimal data.

Methodology: Hierarchical Latent Variable Models

The authors propose modeling system dynamics as a hierarchical latent variable process. The state transition is defined as: $$x_{t+1} = f(x_t, c_t, h_p) + \epsilon$$ where $f$ represents the global transition function (shared physics), $c_t$ is the control input, and $h_p$ is a low-dimensional latent variable representing the specific properties of task $p$.

The graphical model of ML-GP illustrates the hierarchy where global dynamics $f$ and task-specific latent variables $h_p$ combine to generate observations $y_t$ for each task $p$ over time steps $t$.

By using Gaussian Processes (GPs) to represent $f$, the model provides non-parametric flexibility and well-calibrated uncertainty estimates. To handle new tasks online, the framework employs variational inference to update the posterior distribution of $h_p$ as observations arrive. This inferred model is then used within a Model Predictive Control (MPC) loop, where the agent continuously re-plans its trajectory based on its evolving understanding of the task's specific dynamics.

Experimental Evaluation

The ML-GP framework was evaluated on two classic control benchmarks modified for meta-learning:

  • Cart-pole Swing-up: Tasks varied by the mass and length of the pendulum.
  • Double-pendulum Swing-up: A complex two-link system where the physical parameters of the inner pendulum were varied.

Results and Discussion

The ML-GP approach demonstrated significant improvements in data efficiency, reducing average interaction time by up to 60% compared to standard GP baselines. In the Cart-pole environment, ML-GP achieved an 80% success rate within just 3 trials, whereas independent models required more than double that experience.

Comparison of ML-GP against Sparse GP (SGP) and standard GP on a test task. ML-GP (top) accurately tracks the test data by correctly inferring the latent task parameters, while baselines struggle to generalize.

A key insight from the experiments is the model's ability to perform automatic disentanglement. Without explicit supervision, the learned latent space $h_p$ naturally organized itself to reflect physical parameters like mass and length. This structure enables "single-shot" generalization; the model can often identify the correct environment dynamics after observing as little as one second of data (10 time steps). Furthermore, the use of GPs ensures that the agent remains cautious when uncertainty is high, providing a natural safety mechanism during the initial exploration of a new task.

Conclusion

ML-GP provides a robust and data-efficient path for meta-RL by explicitly modeling task-specific variations through latent variables. While the current implementation assumes static task properties and faces scaling challenges inherent to GPs, it represents a major step toward practical robotic learning. Future work may explore deep latent variable models for high-dimensional inputs and extensions to non-stationary environments where task parameters change over time.

Summary updated

Abstract

Learning from small data sets is critical in many practical applications where data collection is time consuming or expensive, e.g., robotics, animal experiments or drug design. Meta learning is one way to increase the data efficiency of learning algorithms by generalizing learned concepts from a set of training tasks to unseen, but related, tasks. Often, this relationship between tasks is hard coded or relies in some other way on human expertise. In this paper, we frame meta learning as a hierarchical latent variable model and infer the relationship between tasks automatically from data. We apply our framework in a model-based reinforcement learning setting and show that our meta-learning model effectively generalizes to novel tasks by identifying how new tasks relate to prior ones from minimal data. This results in up to a 60% reduction in the average interaction time needed to solve tasks compared to strong baselines.
BibTeX
@inproceedings{smundsson2018meta,
  author = {Steindór Sæmundsson and Katja Hofmann and Marc Peter Deisenroth},
  title = {{Meta Reinforcement Learning with Latent Variable Gaussian Processes}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=YXFfuvdAdS},
}
Figure