Deluca - A Differentiable Control Library: Environments, Methods, and Benchmarking

Paula Gradu, John Hallman, Daniel Suo, Alex Yu, Naman Agarwal, Udaya Ghai, Karan Singh, Cyril Zhang, Anirudha Majumdar, Elad HazanarXiv 2021 · arXiv

Introduction

Deluca is an open-source library designed to bring native differentiability to the world of physics simulation and control. By leveraging the power of JAX, Deluca allows researchers to compute gradients directly through environment dynamics. This capability enables massive speedups in training and the application of sophisticated gradient-based control methods that were previously computationally prohibitive in standard "black-box" reinforcement learning (RL) settings.

The Problem: Black-Box Dynamics

In traditional RL, environments are typically treated as black boxes. Popular libraries like OpenAI Gym provide a mechanism to step through an environment, but they do not expose the underlying mathematical transitions. This forces researchers to rely on "zero-order" methods, such as random sampling or finite differences, to estimate gradients. These approaches are notoriously slow, high-variance, and computationally expensive.

Furthermore, many robotics simulations involve "hybrid dynamics" (e.g., a robot foot making contact with the ground), which create mathematical discontinuities. Existing differentiable simulators often struggle with these contacts or lack the standard benchmarking tools required for rigorous research.

The Deluca Framework

Deluca is built on JAX, a library for accelerated linear algebra and automatic differentiation. Its core features include:

  • Native Differentiability: Environments are written to be fully compatible with JAX’s grad function, allowing for the calculation of exact sensitivities of future costs to control inputs.
  • JIT Compilation: By utilizing Just-In-Time (JIT) compilation via jax.lax.scan, Deluca transforms Python code into highly efficient machine code at runtime.
  • Gradient-Based Agents: The library includes advanced control algorithms like iLQR (Iterative Linear Quadratic Regulator), GPC (Gradient Perturbation Control), and various adaptive control methods that explicitly exploit these dynamic gradients.

Performance and Efficiency

The primary advantage of differentiability is efficiency. Having access to the "true" gradient of the dynamics (autograd) is significantly more effective than estimating it. For instance, in a quadrotor trajectory optimization task, autograd required 16x fewer function evaluations than finite differences to achieve the same cost reduction.

Comparison of gradient-based (autograd) vs. gradient-free methods on Planar Quadrotor and Pendulum tasks.

Beyond optimization efficiency, the raw simulation speed is dramatically improved. In head-to-head comparisons on an inverted pendulum task, Deluca (JAX) achieved a per-iteration time of 38 nanoseconds, compared to 54 microseconds for OpenAI Gym (NumPy)—a speedup of over 1000x.

Robustness and Real-World Applications

Deluca opens the door to "non-stochastic control," a framework better suited for handling adversarial perturbations like sudden wind gusts or sensor glitches. The library’s GPC and AdaGPC agents demonstrate significantly lower costs than traditional LQR or $H_\infty$ control when faced with sinusoidal or constant adversarial noise.

Average cost over time for various algorithms under sinusoidal noise in an inverted pendulum environment.

Additionally, Deluca demonstrates utility in complex medical simulations. By training a differentiable neural network on real-world data to simulate a medical ventilator and lung-breath dynamics, researchers were able to train a controller that outperformed industry-standard, hand-tuned PID controllers.

Conclusion

While Deluca currently focuses on a specific subset of physics-based environments and continues to address the challenges of contact-based discontinuities, it represents a significant step toward making differentiable simulation a standard tool in RL. By providing high-performance, natively differentiable environments and agents, Deluca fosters a more efficient and robust ecosystem for control research.

Summary updated

Abstract

We present an open-source library of natively differentiable physics and robotics environments, accompanied by gradient-based control methods and a benchmark-ing suite. The introduced environments allow auto-differentiation through the simulation dynamics, and thereby permit fast training of controllers. The library features several popular environments, including classical control settings from OpenAI Gym. We also provide a novel differentiable environment, based on deep neural networks, that simulates medical ventilation. We give several use-cases of new scientific results obtained using the library. This includes a medical ventilator simulator and controller, an adaptive control method for time-varying linear dynamical systems, and new gradient-based methods for control of linear dynamical systems with adversarial perturbations.
BibTeX
@misc{gradu2021deluca,
  author = {Paula Gradu and John Hallman and Daniel Suo and Alex Yu and Naman Agarwal and Udaya Ghai and Karan Singh and Cyril Zhang and Anirudha Majumdar and Elad Hazan},
  title = {{Deluca - A Differentiable Control Library: Environments, Methods, and Benchmarking}},
  year = {2021},
  eprint = {2102.09968},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.09968},
}
Figure