Safe Pontryagin Differentiable Programming

Wanxin Jin, Shaoshuai Mou, George J. Pappas

Introduction

Safe Pontryagin Differentiable Programming (Safe PDP) is a novel framework designed to guarantee that safety constraints are satisfied during the learning process, not just after it finishes. By combining interior-point methods with differentiable optimal control, Safe PDP allows autonomous systems—like rockets or drones—to learn complex tasks with provable safety and high computational efficiency.

The Challenge of Safe Control

In autonomous systems, "safety" typically means operating within strict boundaries: avoiding obstacles, respecting speed limits, or managing fuel consumption. Most machine learning algorithms for control struggle with two main issues:

  1. At-convergence safety: They only guarantee safety after the model has fully finished training. During the exploratory training phase, the system might easily violate constraints, leading to catastrophic failures in real-world scenarios.
  2. Conservatism: To avoid crashing, algorithms might behave so cautiously that they fail to learn the optimal way to perform a task.

Furthermore, existing safe control methods often struggle with "mixed" constraints (involving both the robot's state and its control inputs) or suffer from severe computational bottlenecks as the planning horizon extends.

The Safe PDP Methodology

Safe PDP builds on the foundation of Pontryagin Differentiable Programming, introducing a robust way to handle constraints without sacrificing differentiability or speed.

  • Interior-Point Barrier Functions: Instead of treating constraints as hard boundaries that confuse optimizers, Safe PDP uses logarithmic barrier functions for inequalities and quadratic penalties for equalities. This transforms a constrained problem into an unconstrained one where the "cost" of approaching a safety boundary approaches infinity.
  • Safe Unconstrained Approximation: The authors prove that this unconstrained approximation is "strictly safe." Any intermediate solution found during the optimization process naturally stays within the feasible, safe region.
  • Efficient Gradient Computation: To make the system differentiable for neural network training, Safe PDP derives an auxiliary control system. The gradient of the control solution is computed by solving a specific Linear-Quadratic Regulator (LQR) problem.
  • Linear Time Complexity: Unlike standard solvers that scale poorly, Safe PDP exploits the sparse structure of control problems to achieve $O(T)$ time complexity relative to the time horizon, making it highly efficient for long-term planning.
Effect of the barrier parameter on constraint satisfaction

Experimental Environments

The framework was evaluated across several high-fidelity simulated environments representing challenging physics-based control tasks.

Overview of evaluated systems, dynamics, and constraints

These environments range from classic benchmarks to highly unstable systems:

  • Cartpole: A standard benchmark for balancing and coordinate control.
  • Two-link Robot Arm: Tests coordinate control and joint limit adherence.
  • 6-DoF Maneuvering Quadrotor: A complex drone model requiring precise 3D navigation.
  • 6-DoF Rocket Powered Landing: Mimics SpaceX-style vertical landings, where strict safety constraints (e.g., impact velocity, tilt angle) are critical.

Key Results and Insights

  • In-Progress Safety: Safe PDP successfully trained neural network policies that reached their goals without ever violating state or input constraints during the learning iterations. You do not have to risk the hardware to teach it a new trick.
  • Superior Scaling: Compared to industry-standard tools like CasADi or other Differentiable MPC frameworks (which often scale at $O(T^2)$ or $O(T^3)$), Safe PDP maintains a steady $O(T)$ computational climb, making it vastly superior for long-horizon tasks.
  • Learning from Experts (Inverse RL): The system successfully performed Inverse Reinforcement Learning. By observing an expert pilot's trajectories, Safe PDP accurately recovered the underlying physics, costs, and safety limits of the system.
  • End-to-End Differentiability: By using smooth barrier functions, the entire control pipeline remains differentiable, bridging the rigorous guarantees of classical control theory with the flexibility of deep learning.

Limitations and Future Work

While powerful, Safe PDP has a few limitations. First, it requires a safe initialization: users must provide at least one initial trajectory that doesn't violate constraints to kick off the process, which can be challenging to find in highly complex environments. Second, the framework exhibits numerical sensitivity to the barrier parameter ($\gamma$). If $\gamma$ is set too small to achieve extreme accuracy, the problem can become numerically unstable for some solvers.

Future work aims to extend the framework to handle stochastic systems with random noise and uncertainty, as well as hybrid systems involving discrete logic (e.g., a robot transitioning between walking and jumping).

Summary updated

Abstract

We propose a Safe Pontryagin Differentiable Programming (Safe PDP) methodology, which establishes a theoretical and algorithmic framework to solve a broad class of safety-critical learning and control tasks---problems that require the guarantee of safety constraint satisfaction at any stage of the learning and control progress. In the spirit of interior-point methods, Safe PDP handles different types of system constraints on states and inputs by incorporating them into the cost or loss through barrier functions. We prove three fundamentals of the proposed Safe PDP: first, both the solution and its gradient in the backward pass can be approximated by solving their more efficient unconstrained counterparts; second, the approximation for both the solution and its gradient can be controlled for arbitrary accuracy by a barrier parameter; and third, importantly, all intermediate results throughout the approximation and optimization strictly respect the constraints, thus guaranteeing safety throughout the entire learning and control process. We demonstrate the capabilities of Safe PDP in solving various safety-critical tasks, including safe policy optimization, safe motion planning, and learning MPCs from demonstrations, on different challenging systems such as 6-DoF maneuvering quadrotor and 6-DoF rocket powered landing.
BibTeX
@inproceedings{jin2021safe,
  author = {Wanxin Jin and Shaoshuai Mou and George J. Pappas},
  title = {{Safe Pontryagin Differentiable Programming}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=wfGbrrWgXDm},
}
Figure