TL;DR
This paper introduces a method to induce sparsity in robot control—essentially teaching robots to use only the necessary "muscles" or thrusters to complete a task. By integrating smooth $L_1$ and Huber regularization into the Differential Dynamic Programming (DDP) framework, the authors enable automatic joint selection for complex humanoids and realistic "on/off" thruster behavior for satellites.
What Problem Does This Paper Address?
In traditional optimal control, researchers typically use $L_2$ regularization (squaring the control inputs) to keep energy consumption low and motions smooth. However, $L_2$ has a "spread the wealth" problem: it encourages every single joint or actuator to move at least a little bit.
This is problematic for two main reasons:
- Satellite Propulsion: Many satellite thrusters are binary (on or off). They cannot effectively produce the tiny, continuous forces that $L_2$ optimization suggests.
- Redundant Robots: For a high-degree-of-freedom (DoF) robot like the NASA Valkyrie (38 joints), moving every joint just to reach for a cup is unnecessary. It creates complex coordination challenges for low-level controllers and wastes energy.
The challenge is that the $L_1$ norm (absolute value), which is the standard tool for inducing sparsity, is non-differentiable at zero. This makes it difficult to use with efficient, gradient-based solvers like DDP that rely on second-order derivatives.
Methods
The authors propose using differentiable approximations of the $L_1$ norm within the DDP cost function. They compare three specific sparsity-inducing penalties:
- Smooth $L_1$: A hybrid that is quadratic (like $L_2$) near zero and linear (like $L_1$) elsewhere.
- Huber Loss: Similar to Smooth $L_1$, but with a different transition behavior controlled by a shape parameter $\beta$.
- Pseudo-Huber Loss: A smooth, curved approximation that uses a square root function to mimic the Huber loss while remaining infinitely differentiable.
By adding these to the DDP objective function, the solver is incentivized to drive unnecessary control inputs exactly to zero. The authors also discuss the importance of two hyperparameters: $\lambda$ (the strength of the penalty) and $\beta$ (the threshold for what counts as "near zero").
Datasets
The paper evaluates the approach on three distinct robotic systems (simulated and physical):
- Cartpole: A canonical 1D underactuated system used to study hyperparameter sensitivity.
- SSL-1300 Satellite Model: A floating rigid body in $SE(3)$ with 18 thrusters, used to demonstrate "bang-bang" (on/off) control.
- NASA Valkyrie Humanoid: A 38-DoF robot used for a reaching task to demonstrate automatic joint selection.
Key Insights and Ideas
- Automatic Joint Selection: In the Valkyrie reaching task, the sparsity-inducing cost naturally "deactivated" joints that weren't essential for the reach. While the standard $L_2$ approach moved 25 joints, the Pseudo-Huber approach moved only 7.
- The Sparsity/Task Trade-off: There is a fundamental trade-off. Increasing sparsity ($\lambda$) makes the motion simpler but can slightly increase the final task error or lead to "artifacts" (rapid fluctuations) if not tuned correctly.
- Better Hardware Tracking: Surprisingly, sparse trajectories often performed better on the physical Valkyrie robot. Because fewer joints were moving, the low-level whole-body controller had less "noise" to manage, resulting in lower tracking error at the end of the motion.
- Convergence Speed: Sparsity comes at a computational cost. Huber losses generally take longer to converge than standard $L_2$ because the gradients are less steep far from zero. Among the sparse options, Pseudo-Huber was the fastest.
Key Results
- Satellite Control: The method successfully discovered thruster-like behavior, producing impulsive bursts of force rather than continuous low-level pressure.
- Valkyrie Reaching: The sparse controller reduced the number of active joints by over 70% compared to the $L_2$ baseline.
- Numerical Stability: The authors demonstrated that by choosing the largest $\beta$ (noise tolerance) the system can handle and then tuning $\lambda$, users can achieve high sparsity without sacrificing numerical stability in the DDP solver.
Limitations and Future Work
- Hyperparameter Tuning: The performance is highly sensitive to $\beta$ and $\lambda$. Currently, these require a grid search or manual tuning to balance sparsity against motion artifacts.
- Hardware Wear and Tear: "Bang-bang" control (sudden on/off) involves high accelerations, which can be taxing on electric motors and gearboxes compared to smooth $L_2$ profiles.
- Future Directions: The authors suggest exploring time optimization (allowing the solver to decide how long a move should take) and developing more robust ways to handle control artifacts through advanced regularization strategies.
Summary updated