Constraints as Observations: A Temperature-Annealed Barrier for Differential Dynamic Programming

Robots plan motions with fast local optimizers that refine one trajectory at a time. These optimizers stall in two well-known ways. Hard constraints such as obstacles and torque limits trap them behind walls they cannot cross even temporarily. Soft penalties let them cross but leave small violations, or make the numerical problem badly conditioned. The current fix for the trapping problem is a hand-designed "tolerant" barrier, a smooth penalty shaped from sigmoid and softplus pieces that lets intermediate iterates pass through obstacles. It works, but its shape and scale are tuned by hand.

We propose to derive that barrier instead of designing it. We treat each constraint as an observed event in the control-as-inference view: planning is posterior inference over trajectories conditioned on "success." A constraint then becomes one more piece of evidence with a likelihood whose sharpness is set by a temperature. Under the Gaussian local policies that maximum-entropy Differential Dynamic Programming (DDP) already produces, this likelihood integrates in closed form. The resulting extra term in the value function is a tolerant barrier whose transition width comes from the temperature and whose scale comes from the local curvature of the cost. Annealing the temperature to zero recovers the hard constraint. A light augmented-Lagrangian correction then removes the residual linearization error. The two-sentence version is this: constraint handling and exploration in local trajectory optimization can come from one inference calculation rather than two hand-tuned mechanisms, and the payoff should be escape from non-convex traps without per-task barrier tuning, while still converging to strictly feasible solutions.

Core Claim

Write $g(x,u)\le 0$ for a constraint at one stage. We model it as a binary observation $C_t$ with probit likelihood $P(C_t=1\mid x,u)=\Phi(-g(x,u)/\varepsilon)$. Here $\Phi$ is the standard normal cumulative distribution function and $\varepsilon>0$ is a width that goes to zero for a hard constraint. In the DDP backward pass the constraint is linearized as $g\approx g_0+G\,\delta u$, where $g_0$ is its value along the current trajectory and $G$ its Jacobian with respect to the control change $\delta u$, at a fixed state deviation. The entropy-regularized policy at that stage is Gaussian with mean $\mu$ and covariance $\Sigma=\alpha Q_{uu}^{-1}$, where $\alpha$ is the entropy temperature and $Q_{uu}$ the control block of the local quadratic model. Marginalizing the constraint likelihood over this policy gives an exact closed form. The stage's soft value function acquires the term

$$ -\alpha \log \Phi\!\left(\frac{-(g_0+G\mu)}{\sqrt{\varepsilon^2+\alpha\, G Q_{uu}^{-1} G^\top}}\right). $$

This term is essentially zero when the constraint holds with margin. Far inside the violated region it grows quadratically, scaled by the policy's spread along the constraint normal. It shifts smoothly between these regimes over a width set by $\varepsilon$ and $\alpha$. As $\alpha,\varepsilon\to 0$ it becomes the indicator of the feasible set. The structure mirrors what Tolerant Discrete Barrier States build by hand: non-vanishing gradients toward the safe set while infeasible, and a sharp boundary at convergence. Here the scale is normalized by $Q_{uu}$ rather than tuned. The policy covariance also enters the denominator, so higher exploration automatically softens the constraint. This is the coupling between exploration and constraint tolerance that the two existing mechanisms treat separately.

Annealing alone gives feasibility only for the linearized constraint. We therefore wrap the final low-temperature phase in the primal-dual augmented Lagrangian multiplier update of Constrained Differential Dynamic Programming: A primal-dual augmented Lagrangian approach and ProxDDP, so that the nonlinear residual is driven to solver tolerance. The claim is falsifiable on three points. The derived barrier should match or beat a tuned tolerant barrier on trap-escape benchmarks. It should do so with one default annealing schedule across tasks. And it should end at the same feasibility tolerance as ProxDDP.

Relation to Prior Work

Three papers supply the method we reuse. Maximum Entropy Differential Dynamic Programming derives Gaussian and mixture-of-Gaussian policies from an entropy-regularized Bellman recursion. Its obstacles enter only as smoothed costs. We keep its Gaussian policy but add constraints as likelihoods, which turns its covariance into an ingredient of the barrier. Generalized Maximum Entropy DDP shows that heavy-tailed Tsallis policies explore at much lower temperatures ($\alpha=0.01$ versus $\alpha=20$ for Shannon). It is likewise unconstrained, and we treat it as a competitor and as a possible later extension of the policy family. The primal-dual DDP line supplies exact feasibility at convergence but remains a single local descent with no exploration mechanism.

Two papers motivate the gap. Tolerant Barrier States shows that hard barriers become stuck at the front of a U-shaped "horseshoe" obstacle, while a sigmoid-plus-softplus barrier state escapes. That design costs roughly twice the per-iteration time, adds states to the system, and has hand-set shape parameters. Our proposal tries to replace that construction with a derived term that adds no states. Constrained DDP Revisited shows that slack-variable formulations converge from starts that violate constraints by 0.3 m on a 2D car. This sets the infeasible-start bar we must meet.

Two papers constrain the design. Constrained Variational Policy Optimization also places constraints inside a variational inference step. It solves a convex program per state over learned critics, so it is model-free and pays both numerical overhead and critic error. We instead get a closed-form stage-wise term from exact models, with derivatives from Pinocchio's analytical derivatives. Physics-informed Gaussian Processes as Linear MPC embeds linear constraints in an inference model for linear systems. We target nonlinear, stage-wise DDP. Finally, Crocoddyl documents that quadratic soft penalties allow minor limit violations. This is the failure mode a naive "soft constraint in maximum-entropy DDP" would inherit.

Research Program

The first step is analytic. We will derive the backward pass with the probit-marginal term for control constraints and for linearized state constraints. Because the term depends on the state deviation through $g_0$ and $G$, it is not quadratic. We expand it to second order about the nominal trajectory, the same approximation barrier-state DDP already makes. We will check that the expansion keeps $Q_{uu}$ positive definite. For several simultaneously active constraints the exact marginal is a multivariate normal orthant probability. We will use a product-of-half-spaces approximation, which is exact for one active constraint. We will measure how much it errs by comparing against Monte Carlo estimates on random polytopes.

The second step is the schedule. Temperature $\alpha$ and width $\varepsilon$ start large and shrink geometrically. Once $\alpha$ falls below a threshold, augmented-Lagrangian multiplier updates switch on. We will try to prove a modest statement: under the standard local assumptions of the primal-dual analysis, the hybrid inherits its local convergence once annealing ends. We will not claim any global guarantee.

The third step is implementation inside ALIGATOR, the C++ library released with ProxDDP. Its multiplier machinery can then be reused and the timing comparison stays honest. An optional extension composes several modes via the log-sum-exp pooling of maximum-entropy DDP. We treat this as secondary and do not claim it as a contribution unless the experiments isolate a benefit.

Baselines

The naive reference point is vanilla DDP with quadratic constraint penalties at a fixed weight, plus the same method with a penalty-continuation schedule. This shows what "just soften it" achieves. It is run with the same horizon, initialization, and iteration cap.

The strongest existing method for escaping traps under constraints is T-DBaS-DDP from Tolerant Barrier States. That paper reports qualitative success on the horseshoe and evaluation over 600 randomized fields with up to 25 obstacles. No reported numerical success rate or cost was found on Lacuna for T-DBaS on these benchmarks, so we will re-run it ourselves. We give T-DBaS a per-task hyperparameter search with a fixed budget of trials. Our method gets the same budget for its schedule, and we also report it with a single cross-task default. Beating T-DBaS means a higher success rate or lower final cost at equal wall-clock time, and no worse feasibility. Matching T-DBaS's tuned performance with our untuned default would count as a meaningful win on its own.

The strongest existing method for feasibility and speed is ProxDDP, which reports under 6 ms per whole-body MPC update on SOLO-12 (ProxDDP) and $10^{-8}$ precision in about 10 iterations on bound-constrained LQR. We run it single-start and multi-start with K initializations at matched total compute. We must equal its final constraint violation, and we must beat its success rate on trap benchmarks without exceeding its compute.

Two ablations isolate the key ingredient. The first removes the policy-covariance term from the denominator, leaving a deterministic probit barrier with the same annealing. This tests whether the inference view adds anything beyond "a log-CDF is a nice smooth barrier." The second disables the augmented-Lagrangian correction, which tests whether annealing alone reaches hard feasibility.

Why not just…?

Why not just run ProxDDP from several initializations and keep the best? This is the most serious threat. On a horseshoe, a handful of starts that cover left and right may be enough. The multi-start ProxDDP baseline at matched compute settles it directly. We expect the advantage to appear in the 25-obstacle randomized fields, where the number of topological routes grows faster than any affordable K, and on infeasible starts. If multi-start matches us there too, the honest conclusion is that our contribution reduces to a tuning-free barrier, and we would report it that way.

Why not just use T-DBaS, or any well-tuned softplus continuation? If a tuned softplus schedule matches our derived term everywhere, the derivation is a nice reinterpretation rather than an improvement. Our evidence must therefore come from the sensitivity sweep. We vary each method's shape and scale parameters over two orders of magnitude and plot success rate, expecting ours to stay flat because $Q_{uu}$ normalization absorbs scaling. The covariance-free ablation tells us whether any edge is due to the inference coupling or merely to the log-CDF shape.

Why not just explore with an unconstrained maximum-entropy or sampling planner, then polish with ProxDDP? This two-stage pipeline uses GME-DDP for its strong low-temperature exploration, then a constrained polish. It is a natural and cheap competitor. Its weakness is that the explorer never sees the constraints, so it can pick a route the polisher cannot make feasible, for example a gap narrower than the torque limits allow. We run this pipeline with the same total iterations. Its failure-to-polish rate on constrained-actuator tasks is the decisive number. Pure sampling-based path-integral control is included only on the low-dimensional tasks. Sampling efficiency is reported to degrade with dimension and horizon (Maximum Entropy DDP), so we do not expect it to compete on the UR10 or SOLO-12 tasks.

Proposed Experiments

The trap-escape suite consists of the horseshoe benchmark and the 600-instance randomized obstacle fields of the tolerant-barrier paper, with a differential-drive robot. It also includes the 2D car and 12-state quadrotor tasks with the infeasible starts used in Constrained DDP Revisited (0.3 m violation). Metrics are success rate, defined as reaching the goal with maximum violation below $10^{-6}$ at termination, final cost, iterations, and wall-clock time. The 2D point mass in a maze, where MME-DDP reports a 94.5% mean-cost reduction over vanilla DDP, serves as a sanity check that we reproduce that regime.

The actuator-limit suite consists of quadrotor flight under thrust bounds and the UR10 obstacle-and-torque tasks from ProxDDP. These are where soft penalties are known to leak violations. Both suites include the sensitivity sweeps described above.

The decisive checks are three. On the randomized fields, our default schedule must beat matched-compute multi-start ProxDDP and match tuned T-DBaS. The covariance-free ablation must perform measurably worse, or the inference framing is not earning its place. Final feasibility must equal ProxDDP's tolerance. If those checks pass, a stretch test is warm-started MPC on SOLO-12 in ALIGATOR against the reported 6 ms budget. Robotarium hardware runs on the horseshoe layout would follow (The Robotarium), whose reported 5 mm sim-to-hardware tracking error makes it a reasonable transfer check.