Probabilistic Differential Dynamic Programming

Yunpeng Pan, Evangelos A. TheodorouNeurIPS 2014 · OpenReview

Overview

Probabilistic Differential Dynamic Programming (PDDP) is a novel data-driven framework that enables robots to learn complex control tasks—such as swinging up a double pendulum—without requiring prior knowledge of the underlying physics equations. By combining Gaussian Processes (GPs) for uncertainty-aware modeling with the computational efficiency of Differential Dynamic Programming (DDP), PDDP achieves a "best of both worlds" outcome. It is nearly as data-efficient as state-of-the-art reinforcement learning methods while being significantly faster to compute.

The Challenge of Unknown Dynamics

In classical robotics, Differential Dynamic Programming (DDP) is a standard method for finding optimal trajectories. While DDP is fast and scales well, it has a major vulnerability: it requires an extremely accurate mathematical model of the robot's dynamics. If the model is even slightly inaccurate, the resulting controller often fails.

In real-world scenarios, modeling complex physical phenomena like friction, aerodynamics, or joint flexibility is incredibly difficult. "Model-free" reinforcement learning can learn these dynamics from scratch but typically requires millions of trial-and-error interactions. Conversely, "model-based" methods like PILCO are highly data-efficient but rely on global non-convex optimization, making them computationally expensive and slow. PDDP seeks a middle ground: learning the model probabilistically from data while retaining the local computational efficiency of DDP.

Probabilistic Differential Dynamic Programming (PDDP)

The PDDP framework operates on three foundational pillars:

  1. Gaussian Process (GP) Dynamics: Instead of relying on fixed physics equations, the system uses GPs to learn dynamics from sampled data. Crucially, GPs do not just predict the next state; they output a probability distribution (mean and variance), allowing the robot to quantify its uncertainty.
  2. Gaussian Belief Space: PDDP plans in "belief space" rather than just a single state $x$. The state is represented as a vector containing both the mean ($\mu$) and the covariance ($\Sigma$) of the robot's position and velocity.
  3. Analytical Jacobians: A major technical contribution of this work is the derivation of analytical partial derivatives for the GP-based dynamics. While previous methods often relied on slow "finite difference" approximations, PDDP calculates these gradients exactly, leading to significantly faster iterations.

The algorithm follows an iterative loop: it samples a few trajectories, updates the GP model, performs a backward pass to find a locally optimal control policy, and then interacts with the physical system again only if the model uncertainty becomes too high.

Experimental Environments

To benchmark the algorithm, the authors utilized simulated physical environments:

  • Cart-Double Inverted Pendulum (CDIP): A highly underactuated system with 6 state dimensions. The objective is to swing two connected poles into an upright position and balance them.
  • Six-Link Robotic Arm: A more complex system with 12 state dimensions, where the goal is to move the arm to a specific reaching target.

PDDP was evaluated against classical DDP (which assumes known physics) and PILCO, the current state-of-the-art for data-efficient reinforcement learning.

Core Innovations

  • Uncertainty-Aware Cost Functions: Because PDDP uses a probabilistic model, its cost function can explicitly penalize high variance. This discourages the robot from attempting high-risk maneuvers in state-space regions where it lacks sufficient data.
  • No Policy Parameterization: Unlike modern RL methods that require defining a policy architecture (like a neural network) and optimizing its weights, PDDP finds a locally optimal, time-varying linear controller directly through the Bellman equation.
  • Efficiency through Locality: By focusing on finding a local optimum around a trajectory rather than a global policy for the entire state space, PDDP avoids the massive computational overhead associated with methods like PILCO.

Performance and Results

PDDP demonstrates strong performance across both benchmark tasks, particularly in balancing computational speed and data efficiency.

CDIP Cost Comparison
  • Computational Speed: PDDP is dramatically faster than PILCO. For the 6-link arm task, PILCO requires iterative non-convex optimization that can take significant time, whereas PDDP computes local controls without an extra optimizer.
  • Data Efficiency: PDDP is nearly as efficient as PILCO, requiring only about 15% to 25% more interactions with the physical system to achieve the same goal. Given the massive reduction in compute time, this is a highly favorable trade-off.
6-Link Arm Cost Comparison
  • Analytical Advantage: The time dedicated to linearizing the dynamics in PDDP is less than 10% of the total iteration time. In contrast, classical DDP implementations that use finite differences spend 70-90% of their time on this step.

Limitations and Future Directions

The authors acknowledge a few constraints in the current PDDP framework:

  • Linear Policy Restriction: PDDP learns a linear feedback policy. While efficient, it is less expressive than the non-linear policies produced by PILCO or deep RL, which might be necessary for highly chaotic systems.
  • GP Scaling: Gaussian Processes typically scale poorly with the number of data points ($O(N^3)$ or $O(N^2)$). For very long-duration learning, "sparse" or "local" GP approximations would be necessary to keep computational costs manageable.
  • Local Optima: Like all DDP-based methods, PDDP finds a local optimum. If the initial random trials are exceptionally poor, the algorithm might converge to a sub-optimal solution.

Summary updated

Abstract

We present a data-driven, probabilistic trajectory optimization framework for systems with unknown dynamics, called Probabilistic Differential Dynamic Programming (PDDP). PDDP takes into account uncertainty explicitly for dynamics models using Gaussian processes (GPs). Based on the second-order local approximation of the value function, PDDP performs Dynamic Programming around a nominal trajectory in Gaussian belief spaces. Different from typical gradient-based policy search methods, PDDP does not require a policy parameterization and learns a locally optimal, time-varying control policy. We demonstrate the effectiveness and efficiency of the proposed algorithm using two nontrivial tasks. Compared with the classical DDP and a state-of-the-art GP-based policy search method, PDDP offers a superior combination of data-efficiency, learning speed, and applicability.
BibTeX
@inproceedings{pan2014probabilistic,
  author = {Yunpeng Pan and Evangelos A. Theodorou},
  title = {{Probabilistic Differential Dynamic Programming}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=Hy4VyP-_-H},
}
Figure