Sparsity-Inducing Optimal Control via Differential Dynamic Programming

Traiko Dinev, Wolfgang Merkt, Vladimir Ivan, Ioannis Havoutis, Sethu Vijayakumar

Abstract

Traditional optimal control often relies on $L_2$ regularization, which tends to distribute control effort across all available actuators. This behavior is suboptimal for systems with binary thrusters or high-degree-of-freedom robots where sparse actuation is preferred. This paper presents a framework for inducing sparsity within Differential Dynamic Programming (DDP) by employing differentiable approximations of the $L_1$ norm, such as Smooth $L_1$, Huber, and Pseudo-Huber losses. We demonstrate that these penalties enable automatic joint selection and realistic thruster behavior across various robotic platforms, including the NASA Valkyrie humanoid and a satellite model.

Introduction

In the field of robotics, optimal control typically minimizes a cost function that includes a quadratic penalty on control inputs ($L_2$ norm). While this ensures smooth motions and low energy consumption, it fails to produce sparse control signals. For satellites equipped with "on/off" thrusters, $L_2$ optimization yields continuous, low-magnitude forces that the hardware cannot replicate. Similarly, for high-degree-of-freedom (DoF) humanoids like the NASA Valkyrie, moving every joint for simple tasks increases control complexity and energy waste.

The $L_1$ norm is the standard mathematical tool for inducing sparsity, but its non-differentiability at zero prevents its direct use in second-order solvers like DDP. This work addresses this gap by integrating smooth, differentiable approximations of the $L_1$ norm into the DDP framework.

Methodology

The core of the proposed approach is the replacement of the standard $L_2$ control penalty with differentiable sparsity-inducing functions. The authors evaluate three primary candidates:

  1. Smooth $L_1$: A hybrid function that is quadratic near the origin and linear elsewhere.
  2. Huber Loss: A robust estimator that transitions from quadratic to linear behavior based on a parameter $\beta$.
  3. Pseudo-Huber Loss: A smooth approximation using a square root function, providing infinite differentiability which is beneficial for gradient-based optimization.
Comparison of various sparsity-inducing cost functions against standard L1 and L2 norms.

These functions incentivize the solver to drive non-essential control inputs exactly to zero. The behavior of these penalties is governed by two key hyperparameters: $\lambda$, which controls the overall strength of the sparsity penalty, and $\beta$, which defines the threshold for "near-zero" behavior.

Experimental Results

The method was validated on three systems: a cartpole, an SSL-1300 satellite model, and the NASA Valkyrie humanoid.

Satellite Propulsion

For the satellite model, the sparsity-inducing costs successfully generated "bang-bang" control profiles. Unlike the continuous signals produced by $L_2$ regularization, the Huber-based penalties resulted in impulsive bursts of force, aligning with the physical constraints of binary thrusters.

Control profiles for different regularization methods showing the 'bang-bang' behavior of sparse penalties compared to L2.

Humanoid Joint Selection

In a reaching task with the 38-DoF Valkyrie robot, the sparse controller automatically identified the minimal set of joints required for the motion. While the $L_2$ baseline activated 25 joints, the Pseudo-Huber approach achieved the same task using only 7 joints—a reduction of over 70%. Interestingly, these sparse trajectories often exhibited better tracking performance on physical hardware because the low-level controller had fewer active degrees of freedom to stabilize.

Discussion and Conclusion

The results highlight a fundamental trade-off between sparsity and task precision. While higher $\lambda$ values increase sparsity, they can introduce motion artifacts or slightly higher terminal errors. Among the tested penalties, the Pseudo-Huber loss offered the best balance of convergence speed and numerical stability. Future research will focus on automating hyperparameter tuning and exploring time-optimal control strategies to further refine hardware compatibility.

Summary updated

BibTeX
@inproceedings{dinev2021sparsity,
  author = {Traiko Dinev and Wolfgang Merkt and Vladimir Ivan and Ioannis Havoutis and Sethu Vijayakumar},
  title = {{Sparsity-Inducing Optimal Control via Differential Dynamic Programming}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9560961},
}
Figure