Meta Reinforcement Learning with Latent Variable Gaussian Processes

Steindór Sæmundsson, Katja Hofmann, Marc Peter Deisenroth

Abstract

Reinforcement learning (RL) often requires extensive interaction data, limiting its applicability in real-world scenarios. This paper presents Meta-Learning with Latent Variable Gaussian Processes (ML-GP), a model-based meta-RL framework designed for high data efficiency. By modeling task-specific dynamics through unobserved latent variables within a hierarchical Gaussian Process framework, ML-GP can rapidly adapt to new environments from minimal observations. Evaluated on continuous control tasks like cart-pole and double-pendulum swing-ups, ML-GP demonstrates up to a 60% reduction in interaction time compared to standard baselines while providing well-calibrated uncertainty estimates for safe planning.

Introduction

Reinforcement Learning (RL) is notoriously data-hungry, often requiring thousands of trials that are impractical in real-world robotics or hardware-sensitive applications. While model-based RL—which learns a simulator of the environment to plan actions—improves efficiency, it is sensitive to model inaccuracies. Meta-learning offers a potential solution by sharing knowledge across related tasks, but existing methods often require massive initial datasets or manual task descriptions. This work introduces ML-GP, a method that automatically infers task relationships from minimal data by treating task variations as unobserved latent variables.

Methods

The ML-GP framework employs a hierarchical Bayesian approach to model system dynamics. Instead of learning a single transition function, the model learns a global function $f(x, a, h)$, where $h$ is a latent variable representing task-specific properties (e.g., a robot's arm length or weight).

Graphical model of the ML-GP framework showing the relationship between states $x_t$, actions $u_t$, observations $y_t$, the global function $f$, and task-specific latent variables $h_p$.

Key components include:

  • Gaussian Processes (GPs): Used to model dynamics due to their excellent uncertainty calibration in small-data regimes.
  • Variational Inference: Enables the agent to maintain and update a posterior belief about the latent variable $h$ online.
  • Sparse GP Approximation: Utilizes "inducing points" to ensure computational scalability as data accumulates.
  • Model Predictive Control (MPC): The agent uses the learned model to plan action sequences that minimize cost over a short horizon, re-planning at each time step to incorporate new observations.

Key Results and Insights

The researchers evaluated ML-GP on cart-pole and double-pendulum swing-up tasks, where physical parameters like mass and length were varied across tasks.

Automatic System Identification

A significant finding is that the latent space automatically learns a meaningful physical structure. Without being explicitly told the mass or length of the pendulums, the model organized the latent variables into a grid that mirrored these physical properties.

Visualization of the learned latent space, where tasks are organized according to their underlying physical parameters (mass and length).

Performance and Data Efficiency

  • Interaction Reduction: In cart-pole tasks, ML-GP achieved high success rates with a 60% reduction in interaction time compared to baselines.
  • Generalization: ML-GP exhibited significantly lower prediction error (RMSE) and better-calibrated uncertainty (NLL) on unseen tasks compared to standard Sparse GPs or independent GPs.
  • Complex Control: On the double-pendulum task, ML-GP reached a 90% success rate in approximately 4 trials, whereas the strongest baseline required 9 trials.

Discussion and Conclusion

ML-GP successfully disentangles universal physical laws from task-specific hardware variations. This allows for "single-shot" transfer, where an agent can observe a few frames of movement and immediately adjust its control strategy.

While highly efficient, the computational cost of GPs remains higher than neural networks, posing challenges for high-dimensional state spaces like raw video. Future work will explore more complex latent structures and real-world applications where hardware wear-and-tear acts as a non-stationary latent variable.

Summary updated

Abstract

Learning from small data sets is critical in many practical applications where data collection is time consuming or expensive, e.g., robotics, animal experiments or drug design. Meta learning is one way to increase the data efficiency of learning algorithms by generalizing learned concepts from a set of training tasks to unseen, but related, tasks. Often, this relationship between tasks is hard coded or relies in some other way on human expertise. In this paper, we frame meta learning as a hierarchical latent variable model and infer the relationship between tasks automatically from data. We apply our framework in a model-based reinforcement learning setting and show that our meta-learning model effectively generalizes to novel tasks by identifying how new tasks relate to prior ones from minimal data. This results in up to a 60% reduction in the average interaction time needed to solve tasks compared to strong baselines.
BibTeX
@inproceedings{smundsson2018meta,
  author = {Steindór Sæmundsson and Katja Hofmann and Marc Peter Deisenroth},
  title = {{Meta Reinforcement Learning with Latent Variable Gaussian Processes}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=YXFfuvdAdS},
}
Figure