Abstract
Traditional optimal control often relies on $L_2$ regularization, which tends to distribute control effort across all available actuators. This behavior is suboptimal for systems with binary thrusters or high-degree-of-freedom robots where sparse actuation is preferred. This paper presents a framework for inducing sparsity within Differential Dynamic Programming (DDP) by employing differentiable approximations of the $L_1$ norm, such as Smooth $L_1$, Huber, and Pseudo-Huber losses. We demonstrate that these penalties enable automatic joint selection and realistic thruster behavior across various robotic platforms, including the NASA Valkyrie humanoid and a satellite model.
Introduction
In the field of robotics, optimal control typically minimizes a cost function that includes a quadratic penalty on control inputs ($L_2$ norm). While this ensures smooth motions and low energy consumption, it fails to produce sparse control signals. For satellites equipped with "on/off" thrusters, $L_2$ optimization yields continuous, low-magnitude forces that the hardware cannot replicate. Similarly, for high-degree-of-freedom (DoF) humanoids like the NASA Valkyrie, moving every joint for simple tasks increases control complexity and energy waste.
The $L_1$ norm is the standard mathematical tool for inducing sparsity, but its non-differentiability at zero prevents its direct use in second-order solvers like DDP. This work addresses this gap by integrating smooth, differentiable approximations of the $L_1$ norm into the DDP framework.
Methodology
The core of the proposed approach is the replacement of the standard $L_2$ control penalty with differentiable sparsity-inducing functions. The authors evaluate three primary candidates:
- Smooth $L_1$: A hybrid function that is quadratic near the origin and linear elsewhere.
- Huber Loss: A robust estimator that transitions from quadratic to linear behavior based on a parameter $\beta$.
- Pseudo-Huber Loss: A smooth approximation using a square root function, providing infinite differentiability which is beneficial for gradient-based optimization.
These functions incentivize the solver to drive non-essential control inputs exactly to zero. The behavior of these penalties is governed by two key hyperparameters: $\lambda$, which controls the overall strength of the sparsity penalty, and $\beta$, which defines the threshold for "near-zero" behavior.
Experimental Results
The method was validated on three systems: a cartpole, an SSL-1300 satellite model, and the NASA Valkyrie humanoid.
Satellite Propulsion
For the satellite model, the sparsity-inducing costs successfully generated "bang-bang" control profiles. Unlike the continuous signals produced by $L_2$ regularization, the Huber-based penalties resulted in impulsive bursts of force, aligning with the physical constraints of binary thrusters.
Humanoid Joint Selection
In a reaching task with the 38-DoF Valkyrie robot, the sparse controller automatically identified the minimal set of joints required for the motion. While the $L_2$ baseline activated 25 joints, the Pseudo-Huber approach achieved the same task using only 7 joints—a reduction of over 70%. Interestingly, these sparse trajectories often exhibited better tracking performance on physical hardware because the low-level controller had fewer active degrees of freedom to stabilize.
Discussion and Conclusion
The results highlight a fundamental trade-off between sparsity and task precision. While higher $\lambda$ values increase sparsity, they can introduce motion artifacts or slightly higher terminal errors. Among the tested penalties, the Pseudo-Huber loss offered the best balance of convergence speed and numerical stability. Future research will focus on automating hyperparameter tuning and exploring time-optimal control strategies to further refine hardware compatibility.
Summary updated