Abstract
Reinforcement learning (RL) often requires extensive interaction data, limiting its applicability in real-world scenarios. This paper presents Meta-Learning with Latent Variable Gaussian Processes (ML-GP), a model-based meta-RL framework designed for high data efficiency. By modeling task-specific dynamics through unobserved latent variables within a hierarchical Gaussian Process framework, ML-GP can rapidly adapt to new environments from minimal observations. Evaluated on continuous control tasks like cart-pole and double-pendulum swing-ups, ML-GP demonstrates up to a 60% reduction in interaction time compared to standard baselines while providing well-calibrated uncertainty estimates for safe planning.
Introduction
Reinforcement Learning (RL) is notoriously data-hungry, often requiring thousands of trials that are impractical in real-world robotics or hardware-sensitive applications. While model-based RL—which learns a simulator of the environment to plan actions—improves efficiency, it is sensitive to model inaccuracies. Meta-learning offers a potential solution by sharing knowledge across related tasks, but existing methods often require massive initial datasets or manual task descriptions. This work introduces ML-GP, a method that automatically infers task relationships from minimal data by treating task variations as unobserved latent variables.
Methods
The ML-GP framework employs a hierarchical Bayesian approach to model system dynamics. Instead of learning a single transition function, the model learns a global function $f(x, a, h)$, where $h$ is a latent variable representing task-specific properties (e.g., a robot's arm length or weight).
Key components include:
- Gaussian Processes (GPs): Used to model dynamics due to their excellent uncertainty calibration in small-data regimes.
- Variational Inference: Enables the agent to maintain and update a posterior belief about the latent variable $h$ online.
- Sparse GP Approximation: Utilizes "inducing points" to ensure computational scalability as data accumulates.
- Model Predictive Control (MPC): The agent uses the learned model to plan action sequences that minimize cost over a short horizon, re-planning at each time step to incorporate new observations.
Key Results and Insights
The researchers evaluated ML-GP on cart-pole and double-pendulum swing-up tasks, where physical parameters like mass and length were varied across tasks.
Automatic System Identification
A significant finding is that the latent space automatically learns a meaningful physical structure. Without being explicitly told the mass or length of the pendulums, the model organized the latent variables into a grid that mirrored these physical properties.
Performance and Data Efficiency
- Interaction Reduction: In cart-pole tasks, ML-GP achieved high success rates with a 60% reduction in interaction time compared to baselines.
- Generalization: ML-GP exhibited significantly lower prediction error (RMSE) and better-calibrated uncertainty (NLL) on unseen tasks compared to standard Sparse GPs or independent GPs.
- Complex Control: On the double-pendulum task, ML-GP reached a 90% success rate in approximately 4 trials, whereas the strongest baseline required 9 trials.
Discussion and Conclusion
ML-GP successfully disentangles universal physical laws from task-specific hardware variations. This allows for "single-shot" transfer, where an agent can observe a few frames of movement and immediately adjust its control strategy.
While highly efficient, the computational cost of GPs remains higher than neural networks, posing challenges for high-dimensional state spaces like raw video. Future work will explore more complex latent structures and real-world applications where hardware wear-and-tear acts as a non-stationary latent variable.
Summary updated