Data-Efficient Task Generalization via Probabilistic Model-Based Meta Reinforcement Learning

Arjun Bhardwaj, Jonas Rothfuss, Bhavya Sukhija, Yarden As, Marco Hutter, Stelian Coros, Andreas Krause

TL;DR

PACOH-RL is a model-based meta-reinforcement learning algorithm that allows robots to adapt to new physical conditions—like different tires or surfaces—using just a few minutes of interaction data. By meta-learning a probabilistic prior over dynamics and planning optimistically under uncertainty, it achieves "positive transfer" even when only a handful of previous tasks are available for training.

What Problem Does This Paper Address?

Reinforcement Learning (RL) has achieved incredible feats in simulation, but the "real world" is a messy place where data is expensive and physics are inconsistent. If you train a robot to drive on carpet, it will likely fail on ice.

Standard Meta-RL aims to solve this by learning how to learn, but most existing methods require hundreds or thousands of different "tasks" (environments) to build a useful prior. In robotics, collecting data for 500 different environments is often impossible. Furthermore, many methods struggle with epistemic uncertainty—the "uncertainty about what we don't know"—leading to poor exploration in new, unfamiliar settings.

Methods

The authors introduce PACOH-RL, which combines three powerful ideas:

  1. Probabilistic Meta-Learning (PACOH): Instead of learning a single "average" model, the algorithm meta-learns a distribution over Bayesian Neural Network (BNN) priors. This is done using a PAC-Bayesian framework that provides principled regularization, preventing the model from overfitting when only a few tasks are available.
  2. Stein Variational Gradient Descent (SVGD): To handle the complex math of BNNs, the authors use SVGD to represent the posterior distribution as a set of "particles" (individual neural networks). This allows for efficient approximate inference.
  3. Optimistic Exploration (H-UCRL): When the robot encounters a new task, it doesn't just act greedily. It uses Hallucinated Upper-Confidence Reinforcement Learning. It "hallucinates" that uncertain regions of the environment might yield high rewards, which naturally drives the robot to explore areas where the dynamics model needs the most improvement.

The framework is flexible: it can use a Model Predictive Control (MPC) solver (like iCEM) for real-time planning or a policy-based method (like Soft Actor-Critic) for faster execution.

Datasets

The researchers evaluated PACOH-RL across several benchmarks:

  • Classic Control: Pendulum, Cartpole, and Half-Cheetah (from OpenAI Gym), modified with varying physical parameters (mass, length, etc.) to create different tasks.
  • Sparse Reward Tasks: "Sparse" versions of Pendulum, Cartpole, and a Pusher task where the agent only gets a reward upon reaching a specific goal.
  • Simulated RC Car: A high-fidelity simulator based on realistic autonomous racing dynamics.
  • Real-World Hardware: A custom high-torque remote-controlled (RC) car. For meta-training, they used only 5 tasks (different tire and weight configurations) with just 4 minutes of data per task.

Key Insights and Ideas

  • The "Low-Task" Regime: Most Meta-RL algorithms (like GrBAL/MAML) actually perform worse than standard RL when given very few training tasks because they overfit. PACOH-RL’s Bayesian regularization allows it to achieve "positive transfer" with as few as 5-20 tasks.
  • Uncertainty as a Guide: In environments with sparse rewards, acting "greedily" usually leads to failure because the agent never finds the goal. By planning optimistically based on the BNN’s uncertainty, PACOH-RL explores the state space in a directed manner, significantly outperforming non-optimistic baselines.
  • Efficiency on Hardware: The real-world RC car experiments showed the agent converging to an almost optimal parking policy in less than 2 minutes of real-world interaction.

Key Results

  • Sample Efficiency: In simulation, PACOH-RL reached high reward levels in roughly 5-10 episodes, while non-meta baselines (like PETS or H-UCRL) often required 20+ episodes to reach the same performance.
  • Robustness: In sparse reward environments, PACOH-RL was often the only algorithm to consistently solve the task, whereas greedy versions (PACOH-RL Greedy) or MAML-based methods (GrBAL) frequently stayed at zero reward.
  • Real-World Success: On the physical RC car, PACOH-RL systematically outperformed PETS-DL, demonstrating that the meta-learned prior provides a much better starting point for adaptation than learning from scratch.

Limitations and Future Work

  • Computational Overhead: Planning with a BNN ensemble is computationally expensive. Table V shows that "Planning" accounts for 70-90% of the total compute time, which could be a bottleneck for high-frequency control on smaller embedded systems.
  • Dimensionality: While it works well for 6D state spaces (like the RC car), the authors note that MPC-based solvers can struggle with the "curse of dimensionality" in much higher-dimensional spaces (e.g., complex humanoid robots).
  • Safety: While the model explores "optimistically," it doesn't explicitly account for safety constraints. Future work could involve integrating "Safe RL" to ensure the robot doesn't damage itself while exploring uncertain dynamics.

Summary updated

Abstract

We introduce PACOH-RL, a novel model-based Meta-Reinforcement Learning (Meta-RL) algorithm designed to efficiently adapt control policies to changing dynamics. PACOH-RL meta-learns priors for the dynamics model, allowing swift adaptation to new dynamics with minimal interaction data. Existing Meta-RL methods require abundant meta-learning data, limiting their applicability in settings such as robotics, where data is costly to obtain. To address this, PACOH-RL incorporates regularization and epistemic uncertainty quantification in both the meta-learning and task adaptation stages. When facing new dynamics, we use these uncertainty estimates to effectively guide exploration and data collection. Overall, this enables positive transfer, even when access to data from prior tasks or dynamic settings is severely limited. Our experiment results demonstrate that PACOH-RL outperforms model-based RL and model-based Meta-RL baselines in adapting to new dynamic conditions. Finally, on a real robotic car, we showcase the potential for efficient RL policy adaptation in diverse, data-scarce conditions.
BibTeX
@article{bhardwaj2024data,
  author = {Arjun Bhardwaj and Jonas Rothfuss and Bhavya Sukhija and Yarden As and Marco Hutter and Stelian Coros and Andreas Krause},
  title = {{Data-Efficient Task Generalization via Probabilistic Model-Based Meta Reinforcement Learning}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  doi = {10.1109/lra.2024.3371260},
}
Figure