Gaussian Processes for Data-Efficient Learning in Robotics and Control

Marc Peter Deisenroth, Dieter Fox, Carl Edward Rasmussen

Abstract

Autonomous learning has been a promising direction in control and robotics for more than a decade since data-driven learning allows to reduce the amount of engineering knowledge, which is otherwise required. However, autonomous reinforcement learning (RL) approaches typically require many interactions with the system to learn controllers, which is a practical limitation in real systems, such as robots, where many interactions can be impractical and time consuming. To address this problem, current learning approaches typically require task-specific knowledge in form of expert demonstrations, realistic simulators, pre-shaped policies, or specific knowledge about the underlying dynamics. In this article, we follow a different approach and speed up learning by extracting more information from data. In particular, we learn a probabilistic, non-parametric Gaussian process transition model of the system. By explicitly incorporating model uncertainty into long-term planning and controller learning our approach reduces the effects of model errors, a key problem in model-based learning. Compared to state-of-the art RL our model-based policy search method achieves an unprecedented speed of learning. We demonstrate its applicability to autonomous learning in real robot and control tasks.

Introduction

Reinforcement Learning (RL) offers a path toward autonomous robots that learn from experience. However, a significant barrier to practical deployment is the "data hunger" of most RL algorithms, which often require thousands of trials to master simple tasks. For physical hardware, this leads to excessive wear, power consumption, and time requirements.

While model-based RL—where an agent learns a simulator of its environment—is generally more efficient than model-free alternatives, it is plagued by model errors. If an agent treats an inaccurate learned model as ground truth, it optimizes for a "fake" world, leading to failure on real hardware. To mitigate this, researchers often resort to hand-coded simulators or expert demonstrations. This paper introduces PILCO (Probabilistic Inference for Learning Control), a framework that achieves unprecedented data efficiency by explicitly modeling and planning with uncertainty.

Methodology

PILCO is designed to extract maximum information from limited data through several key technical components.

Gaussian Process Dynamics

Instead of a deterministic transition function, PILCO employs Gaussian Processes (GPs) to model system dynamics. As a non-parametric Bayesian approach, GPs provide not only state predictions but also a formal measure of uncertainty. This allows the model to express "how much it doesn't know" about specific regions of the state space.

Gaussian Process modeling of system dynamics. Left: Observed data points. Middle: Sample functions from the GP posterior. Right: The GP posterior mean and 95% confidence interval.

Uncertainty Propagation and Moment Matching

A core innovation of PILCO is how it handles long-term planning. Rather than projecting only the most likely future state, PILCO propagates the entire probability distribution of the robot's state through the GP model. Because mapping a Gaussian through a GP results in a non-Gaussian distribution, PILCO uses moment matching to analytically compute the mean and covariance of the state at each future time step. This "Bayesian averaging" ensures that the policy remains cautious in uncertain regions.

Uncertainty propagation in PILCO. The input distribution is mapped through the GP model to find the distribution of the state change. Moment matching (blue) provides a superior Gaussian approximation of the ground truth (green) compared to linearization (red).

Policy Optimization

The entire pipeline—from GP prediction to cost evaluation—is mathematically differentiable. This allows PILCO to compute analytic policy gradients, which are significantly more efficient for optimization than sampling-based "guess-and-check" methods. Additionally, the use of saturating cost functions encourages exploration when far from the goal and exploitation when close.

Experimental Evaluation

The framework was evaluated on both simulated benchmarks and real-world hardware:

  • Cart-Pole Swing-Up: In simulation, PILCO mastered the task in just 17.5 seconds of interaction time.
  • Robotic Unicycle: In a complex 12D state space simulation, the system learned to balance in approximately 30 seconds.
  • Real-World Manipulation: A low-cost robotic arm learned to stack foam blocks using visual feedback from a depth camera in just 10 iterations (roughly 4 minutes of interaction).

Discussion and Insights

The success of PILCO stems from its treatment of model uncertainty as a first-class citizen. By planning for uncertainty, the robot naturally avoids risky maneuvers in unexplored areas, effectively smoothing the cost landscape and avoiding local minima. This approach allows for learning rates 100 to 1000 times faster than state-of-the-art model-free methods.

However, challenges remain. The $O(n^3)$ computational complexity of GPs limits the framework to tasks learnable within a few thousand data points. Furthermore, the moment-matching approach assumes a unimodal (Gaussian) state distribution, which may fail in scenarios requiring multi-modal "fork-in-the-road" decision-making.

Conclusion

PILCO demonstrates that by rigorously accounting for model uncertainty, reinforcement learning can reach the levels of data efficiency required for real-world robotics. By moving away from deterministic models and toward probabilistic, non-parametric representations, we can enable robots to learn complex control behaviors in minutes rather than hours or days.

Summary updated

Abstract

Autonomous learning has been a promising direction in control and robotics for more than a decade since data-driven learning allows to reduce the amount of engineering knowledge, which is otherwise required. However, autonomous reinforcement learning (RL) approaches typically require many interactions with the system to learn controllers, which is a practical limitation in real systems, such as robots, where many interactions can be impractical and time consuming. To address this problem, current learning approaches typically require task-specific knowledge in form of expert demonstrations, realistic simulators, pre-shaped policies, or specific knowledge about the underlying dynamics. In this article, we follow a different approach and speed up learning by extracting more information from data. In particular, we learn a probabilistic, non-parametric Gaussian process transition model of the system. By explicitly incorporating model uncertainty into long-term planning and controller learning our approach reduces the effects of model errors, a key problem in model-based learning. Compared to state-of-the art RL our model-based policy search method achieves an unprecedented speed of learning. We demonstrate its applicability to autonomous learning in real robot and control tasks.
BibTeX
@article{deisenroth2013gaussian,
  author = {Marc Peter Deisenroth and Dieter Fox and Carl Edward Rasmussen},
  title = {{Gaussian Processes for Data-Efficient Learning in Robotics and Control}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2013},
  doi = {10.1109/tpami.2013.218},
}
Figure