Leveraging randomized smoothing for optimal control of nonsmooth dynamical systems

Quentin Le Lidec, Louis Montaut, Ivan Laptev, Justin CarpentierNonlinear 2024 · DOI

Introduction

Optimal control in robotics often involves navigating complex, nonsmooth physical phenomena such as friction and contact. While classical algorithms like Differential Dynamic Programming (DDP) offer high speed by utilizing gradients, they frequently fail when encountering the "jagged" optimization landscapes typical of real-world physics. Conversely, Reinforcement Learning (RL) provides robustness through stochastic exploration but suffers from extreme sample inefficiency. This paper introduces Randomized Differential Dynamic Programming (RDDP), a hybrid approach that integrates randomized smoothing into the DDP framework to achieve both robustness and efficiency.

The Challenge of Nonsmooth Dynamics

In robotic tasks, such as a foot striking the ground or a block initiating a slide, the underlying physics is often non-smooth. These events create sharp discontinuities or "flat" regions where gradients are zero or non-informative. Standard gradient-based solvers like DDP rely on the "slope" of the physics to find optimal trajectories; when these slopes disappear or change abruptly, the solvers become trapped in local optima or fail to converge.

Randomized Differential Dynamic Programming (RDDP)

The core innovation of RDDP is the application of Randomized Smoothing (RS) within the recursive structure of DDP. Instead of calculating the gradient of a non-smooth function $f(u)$ directly, the algorithm evaluates the expected behavior in a local neighborhood:

$$f_{\epsilon}(u) = \mathbb{E}[f(u + \epsilon Z)]$$

This process effectively "blurs" sharp edges, rendering the function differentiable everywhere and providing informative gradients even in contact-rich scenarios.

Key Algorithmic Features

  • Control-Space Smoothing: By smoothing only the control space, RDDP maintains a low sample count while capturing the essential dynamics.
  • Adaptive Noise Scheduling: The algorithm employs an auto-tuning strategy, starting with high noise ($\epsilon$) to explore the global landscape and gradually reducing it to zero to ensure a precise, deterministic final trajectory.
  • Efficient Estimation: Using Monte-Carlo sampling, the authors demonstrate that as few as 8 samples per time step are sufficient to provide the gradient information necessary for complex tasks.

Experimental Evaluation

The RDDP algorithm was evaluated on several challenging robotic benchmarks using the Pinocchio dynamics library and the Crocoddyl framework. These included an inverted pendulum, cube lifting/sliding, cartpole with dry friction, and a quadrotor takeoff.

The Solo robot leg benchmark, illustrating the application of perturbation torques within the RDDP framework to navigate nonsmooth contact dynamics.

A particularly complex test involved the Solo robot leg, a jumping task requiring intricate contact planning. Unlike traditional methods that require pre-defined contact schedules, RDDP allows the solver to "discover" optimal contact phases through smoothed gradients.

Results and Discussion

RDDP demonstrated significant improvements over classical DDP and standard RL:

  • Robustness: In tasks like quadrotor takeoff and cube lifting—where initial gradients are zero—classical DDP fails immediately. RDDP successfully solved these tasks within 10–20 iterations.
  • Sample Efficiency: RDDP achieves high-quality results with only 4 to 8 samples per time step, a massive reduction compared to the millions of samples typically required by RL.
  • Convergence Precision: The adaptive noise schedule allows RDDP to converge to a high-precision solution (near-zero gradient norm), surpassing the performance of RL algorithms that often struggle with a persistent "noise floor."

Conclusion

RDDP bridges the gap between structured optimal control and stochastic reinforcement learning. While it introduces some computational overhead compared to standard DDP, its ability to handle nonsmooth dynamics with high sample efficiency marks a significant step forward. Future work will explore the use of variance information for risk-aware control and scaling the approach to high-degree-of-freedom systems like humanoids.

Summary updated

BibTeX
@inproceedings{lidec2024leveraging,
  author = {Quentin Le Lidec and Louis Montaut and Louis Montaut and Ivan Laptev and Justin Carpentier and Justin Carpentier},
  title = {{Leveraging randomized smoothing for optimal control of nonsmooth dynamical systems}},
  booktitle = {Nonlinear},
  year = {2024},
  doi = {10.1016/j.nahs.2024.101468},
}
Figure