Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control

Sanket Kamthe, Marc Peter Deisenroth

Abstract

Trial-and-error based reinforcement learning (RL) often requires a prohibitive number of interactions with the environment, making it impractical for real-world robotics where safety and hardware wear are major concerns. We propose a model-based RL framework, GP-MPC, which combines Gaussian Processes (GPs) for learning probabilistic transition models with Model Predictive Control (MPC) for constrained long-term planning. By using moment matching to propagate uncertainty, we transform a stochastic control problem into a deterministic one in the space of probability distributions, allowing the application of Pontryagin’s Maximum Principle for efficient optimization. Our approach achieves state-of-the-art data efficiency and provides a principled method for handling state and control constraints.

Introduction: The Data Efficiency Challenge

Reinforcement learning has achieved remarkable success in simulated environments, yet its application to physical systems remains limited by high sample complexity. Standard RL algorithms often require thousands of trials to master simple tasks. In robotics, this "data hunger" is a significant barrier: physical hardware is subject to wear, and unconstrained trial-and-error can lead to dangerous collisions.

While model-based approaches like PILCO have significantly improved data efficiency by learning dynamics models, they often rely on fixed global policies and struggle with hard state constraints (e.g., ensuring a robot arm avoids a wall). The proposed GP-MPC framework addresses these issues by integrating probabilistic dynamics with the robust constraint-handling capabilities of Model Predictive Control.

Methodology: GP-MPC

The GP-MPC framework bridges the gap between probabilistic machine learning and classical optimal control through four key components:

  1. Probabilistic Dynamics with GPs: Instead of a deterministic model, we use Gaussian Processes to learn the environment's transition function. GPs provide a principled way to represent model uncertainty, which is crucial for avoiding overconfidence in data-sparse regions.
  2. Uncertainty Propagation via Moment Matching: To plan over a horizon, the agent must predict future states. We propagate the state distribution (mean and covariance) through the GP using moment matching. This approximation effectively turns the stochastic system into a deterministic mapping between probability distributions.
  3. Optimization via Pontryagin’s Maximum Principle (PMP): Because the evolution of the state distribution is now deterministic, we can apply PMP. This classical control tool provides necessary conditions for optimality, enabling efficient gradient-based optimization of control sequences while strictly respecting constraints.
  4. Receding Horizon Control: Unlike global policy search, GP-MPC solves for an optimal action sequence from the current state, executes the first action, and then re-plans. This "closed-loop" behavior makes the system more robust to model inaccuracies.

Experimental Setup

The authors evaluated GP-MPC on two continuous control benchmarks modified to include physical constraints:

  • Under-actuated Cart-Pole Swing-Up: The agent must swing a pendulum to an upright position while avoiding a "wall" constraint on the track.
  • Fully-actuated Double-Pendulum Swing-Up: A two-link arm task where the inner joint is restricted to a 340° range.
Figure 2: Experimental benchmarks with constraints: (a) Cart-pole with a wall constraint, (b) Double pendulum with joint limits.

Key Results and Discussion

GP-MPC demonstrates superior performance in both data efficiency and safety compared to existing state-of-the-art methods.

State-of-the-Art Data Efficiency

In the double-pendulum task, GP-MPC reached a 90% success rate in just 18 seconds of interaction time (approximately 6 trials), requiring significantly less data than the PILCO algorithm. The use of MPC allows the agent to correct for model errors at every timestep, leading to faster convergence than global policy search methods.

Figure 5: Success rates over trials for (a) Under-actuated cart-pole and (b) Double pendulum. GP-MPC (blue) consistently outperforms PILCO (red) and the Zero-Variance baseline (yellow).

Principled Constraint Handling

By incorporating predictive variance into the optimization process (using "chance constraints"), GP-MPC ensures safer learning. In constrained environments, versions of the algorithm that ignored uncertainty (such as the "Zero-Var" baseline) often failed to learn or frequently violated safety limits. GP-MPC's integration of PMP allows it to treat constraints as fundamental requirements rather than mere suggestions in the cost function, leading to much safer learning in the early stages.

Conclusion

GP-MPC offers a robust framework for reinforcement learning in environments where data is scarce and safety is paramount. By treating the evolution of uncertainty as a deterministic process, the method leverages the strengths of both Gaussian Processes and classical optimal control. While computational complexity remains a challenge for scaling to very high-dimensional inputs, GP-MPC sets a new standard for data-efficient, constrained RL in robotics.

Summary updated

Abstract

Trial-and-error based reinforcement learning (RL) has seen rapid advancements in recent times, especially with the advent of deep neural networks. However, the majority of autonomous RL algorithms require a large number of interactions with the environment. A large number of interactions may be impractical in many real-world applications, such as robotics, and many practical systems have to obey limitations in the form of state space or control constraints. To reduce the number of system interactions while simultaneously handling constraints, we propose a model-based RL framework based on probabilistic Model Predictive Control (MPC). In particular, we propose to learn a probabilistic transition model using Gaussian Processes (GPs) to incorporate model uncertainty into long-term predictions, thereby, reducing the impact of model errors. We then use MPC to find a control sequence that minimises the expected long-term cost. We provide theoretical guarantees for first-order optimality in the GP-based transition models with deterministic approximate inference for long-term planning. We demonstrate that our approach does not only achieve state-of-the-art data efficiency, but also is a principled way for RL in constrained environments.
BibTeX
@inproceedings{kamthe2018data,
  author = {Sanket Kamthe and Marc Peter Deisenroth},
  title = {{Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2018},
  url = {https://openreview.net/forum?id=VZbQAeuL87},
}
Figure