Introduction
Deluca is an open-source library designed to bring native differentiability to the world of physics simulation and control. By leveraging the power of JAX, Deluca allows researchers to compute gradients directly through environment dynamics. This capability enables massive speedups in training and the application of sophisticated gradient-based control methods that were previously computationally prohibitive in standard "black-box" reinforcement learning (RL) settings.
The Problem: Black-Box Dynamics
In traditional RL, environments are typically treated as black boxes. Popular libraries like OpenAI Gym provide a mechanism to step through an environment, but they do not expose the underlying mathematical transitions. This forces researchers to rely on "zero-order" methods, such as random sampling or finite differences, to estimate gradients. These approaches are notoriously slow, high-variance, and computationally expensive.
Furthermore, many robotics simulations involve "hybrid dynamics" (e.g., a robot foot making contact with the ground), which create mathematical discontinuities. Existing differentiable simulators often struggle with these contacts or lack the standard benchmarking tools required for rigorous research.
The Deluca Framework
Deluca is built on JAX, a library for accelerated linear algebra and automatic differentiation. Its core features include:
- Native Differentiability: Environments are written to be fully compatible with JAX’s
gradfunction, allowing for the calculation of exact sensitivities of future costs to control inputs. - JIT Compilation: By utilizing Just-In-Time (JIT) compilation via
jax.lax.scan, Deluca transforms Python code into highly efficient machine code at runtime. - Gradient-Based Agents: The library includes advanced control algorithms like iLQR (Iterative Linear Quadratic Regulator), GPC (Gradient Perturbation Control), and various adaptive control methods that explicitly exploit these dynamic gradients.
Performance and Efficiency
The primary advantage of differentiability is efficiency. Having access to the "true" gradient of the dynamics (autograd) is significantly more effective than estimating it. For instance, in a quadrotor trajectory optimization task, autograd required 16x fewer function evaluations than finite differences to achieve the same cost reduction.
Beyond optimization efficiency, the raw simulation speed is dramatically improved. In head-to-head comparisons on an inverted pendulum task, Deluca (JAX) achieved a per-iteration time of 38 nanoseconds, compared to 54 microseconds for OpenAI Gym (NumPy)—a speedup of over 1000x.
Robustness and Real-World Applications
Deluca opens the door to "non-stochastic control," a framework better suited for handling adversarial perturbations like sudden wind gusts or sensor glitches. The library’s GPC and AdaGPC agents demonstrate significantly lower costs than traditional LQR or $H_\infty$ control when faced with sinusoidal or constant adversarial noise.
Additionally, Deluca demonstrates utility in complex medical simulations. By training a differentiable neural network on real-world data to simulate a medical ventilator and lung-breath dynamics, researchers were able to train a controller that outperformed industry-standard, hand-tuned PID controllers.
Conclusion
While Deluca currently focuses on a specific subset of physics-based environments and continues to address the challenges of contact-based discontinuities, it represents a significant step toward making differentiable simulation a standard tool in RL. By providing high-performance, natively differentiable environments and agents, Deluca fosters a more efficient and robust ecosystem for control research.
Summary updated