Gaussian Processes for Data-Efficient Learning in Robotics and Control

Marc Peter Deisenroth, Dieter Fox, Carl Edward Rasmussen

Abstract

Autonomous reinforcement learning (RL) typically requires extensive interaction with physical systems, which is often impractical in robotics. We present PILCO (Probabilistic Inference for Learning Control), a model-based policy search framework that significantly improves data efficiency. By learning a probabilistic, non-parametric Gaussian process (GP) transition model and explicitly incorporating model uncertainty into long-term planning, PILCO reduces the impact of model errors. Our approach achieves unprecedented learning speeds, enabling real robots to master complex control tasks in minutes rather than hours.

Introduction

Reinforcement learning has demonstrated remarkable success in virtual environments, yet its application to physical robotics remains constrained by high data requirements. Traditional model-free methods often necessitate millions of trials, a "data-hungry" nature that leads to hardware wear and prohibitive time costs. Model-based RL offers a potential solution by learning a "world model" to simulate trajectories; however, it frequently suffers from model bias. When an agent optimizes a policy against a deterministic model that contains even minor inaccuracies, the resulting policy often fails in the real world—a phenomenon known as over-optimization of model errors.

The PILCO framework addresses this by being "honest" about model uncertainty. Instead of relying on expert demonstrations or hand-tuned simulators, PILCO learns from scratch by accounting for what it does not know, using Gaussian Processes to represent the underlying system dynamics.

Probabilistic Dynamics Modeling

At the core of PILCO is a non-parametric Gaussian Process (GP) transition model. Unlike deterministic models that predict a single point estimate for the next state, a GP provides a full posterior distribution.

GP modeling of dynamics showing data points, function samples, and uncertainty bounds.

As shown above, the GP uses available data points to infer a range of plausible functions. In regions with sparse data, the model's uncertainty (variance) increases. This uncertainty is not ignored; it is propagated through the planning process, acting as a natural regularizer that prevents the policy from exploiting inaccuracies in the learned model.

To evaluate a policy over a long time horizon, PILCO must predict the distribution of future states. Because the GP model is non-linear, passing a Gaussian state distribution through it results in a complex, non-Gaussian distribution. PILCO employs Moment Matching to analytically approximate these future distributions as Gaussians.

Comparison of moment matching and linearization against ground truth.

While simpler methods like linearization are computationally cheaper, they often significantly underestimate uncertainty (as seen in the figure above). Moment matching provides a more robust approximation of the ground truth, ensuring that the policy search accounts for the full breadth of potential outcomes. Furthermore, because the entire process—from GP inference to the cost function—is differentiable, PILCO can compute analytic policy gradients, enabling the use of efficient gradient-based optimization algorithms like BFGS.

Experimental Evaluation

The framework was evaluated on both simulated benchmarks and real-world hardware:

  • Cart-Pole Swing-Up: In a real-world setting with friction and sensor delays, PILCO learned a successful policy in just 17.5 seconds of interaction time.
  • Robotic Unicycle: PILCO successfully balanced a simulated 12-dimensional, highly unstable unicycle system in approximately 30 seconds of experience.
  • Robotic Arm Stacking: Using a low-cost Lynxmotion arm and a depth camera, the system learned to stack blocks in roughly 50 seconds per block, successfully compensating for jerky servo movements and visual noise.

Key Insights and Conclusion

PILCO’s efficiency stems from its treatment of uncertainty as a "shield." By using a saturating cost function, the agent is naturally penalized for entering regions where the model is uncertain, which implicitly encourages safe exploration. Once the model is sufficiently accurate near the target, the agent converges rapidly.

While PILCO represents a major step forward in data-efficient RL, challenges remain regarding the $O(N^3)$ computational scaling of GPs and the assumption of independent noise over time. Future work will focus on scaling to higher-dimensional state spaces and adapting the framework for non-episodic, life-long learning scenarios.

Summary updated

Abstract

Autonomous learning has been a promising direction in control and robotics for more than a decade since data-driven learning allows to reduce the amount of engineering knowledge, which is otherwise required. However, autonomous reinforcement learning (RL) approaches typically require many interactions with the system to learn controllers, which is a practical limitation in real systems, such as robots, where many interactions can be impractical and time consuming. To address this problem, current learning approaches typically require task-specific knowledge in form of expert demonstrations, realistic simulators, pre-shaped policies, or specific knowledge about the underlying dynamics. In this paper, we follow a different approach and speed up learning by extracting more information from data. In particular, we learn a probabilistic, non-parametric Gaussian process transition model of the system. By explicitly incorporating model uncertainty into long-term planning and controller learning our approach reduces the effects of model errors, a key problem in model-based learning. Compared to state-of-the art RL our model-based policy search method achieves an unprecedented speed of learning. We demonstrate its applicability to autonomous learning in real robot and control tasks.
BibTeX
@article{deisenroth2013gaussian,
  author = {Marc Peter Deisenroth and Dieter Fox and Carl Edward Rasmussen},
  title = {{Gaussian Processes for Data-Efficient Learning in Robotics and Control}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2013},
  doi = {10.1109/tpami.2013.218},
}
Figure