TL;DR
DEPS (Direct Environment and Policy Search) is a new deep reinforcement learning algorithm that jointly optimizes a system's physical design and its control policy. By leveraging differentiable models and projected gradient ascent, DEPS achieves higher performance with better sample efficiency and stability than existing state-of-the-art methods across diverse engineering benchmarks.
What Problem Does This Paper Address?
In traditional engineering, design and control are often treated as sequential steps: first, you build the hardware (the "environment"), then you write the software to control it (the "policy"). However, these two aspects are deeply intertwined. For example, a drone's arm length affects both its structural stability and its maneuverability. Designing them separately often leads to sub-optimal performance.
The challenge lies in the complexity of the joint optimization space. The objective function—maximizing expected rewards over time—is typically non-convex, high-dimensional, and subject to uncertainty. Previous attempts to solve this used either derivative-free methods (which are slow and scale poorly) or complex reinforcement learning (RL) approaches that optimize a distribution over designs, which can be noisy and unstable.
Methods
The authors introduce Direct Environment and Policy Search (DEPS). The core idea is to treat the physical design parameters ($\psi$) and the control policy parameters ($\theta$) as a single vector to be optimized simultaneously.
Key technical components include:
- Differentiable Programming: DEPS assumes that the system's transition function (physics) and reward function are known and differentiable. This allows the use of automatic differentiation (autodiff) to compute exact gradients.
- Projected Stochastic Gradient Ascent: The algorithm uses Monte-Carlo sampling to estimate the gradient of the expected return. It then takes a step in the direction of the gradient and "projects" the result back into the feasible parameter space (e.g., ensuring a battery capacity isn't negative).
- Direct Parameter Search: Unlike previous methods like JODC that learn a probability distribution over designs, DEPS optimizes the design parameters directly. This significantly reduces variance and improves convergence speed.
Datasets
The authors evaluated DEPS on three distinct, increasingly complex simulated environments:
- Mass-Spring-Damper (MSD): A classic physics benchmark where the goal is to stabilize a mass at a specific position by optimizing spring constants and damping coefficients alongside the controller.
- Off-Grid Microgrid: A socio-technical system where the agent must size a battery, solar panels, and a diesel generator to minimize the total cost of energy over a lifetime.
- Drone Design: A high-dimensional robotics task where the agent optimizes the arm length, thickness, and propeller radius of a quadrotor to fly an elliptical path as fast as possible despite wind gusts.
The implementation and benchmarks are available on GitHub.
Key Insights and Ideas
- The "Direct" Advantage: By optimizing design parameters directly rather than learning a distribution over them, DEPS avoids the high variance that plagues distributional methods. This makes the learning process much more stable.
- Hardware as a Differentiable Layer: The paper treats the physical environment as just another layer in a neural network. If you can differentiate through the physics, you can use the same tools (like Adam or SGD) used to train AI to design hardware.
- Co-dependency is Crucial: The experiments show that a policy optimized for one design performs poorly if the design changes slightly. Joint optimization finds a "sweet spot" where the hardware's physical properties actually make the control task easier.
- Sample Efficiency: Because DEPS uses the analytical form of the environment to compute gradients, it requires significantly fewer simulations to reach an optimal solution compared to model-free or distributional RL methods.
Key Results
- Superior Performance: In all three environments, DEPS achieved higher expected returns than the JODC (Joint Optimization of Design and Control) baseline.
- Stability: DEPS showed much tighter confidence intervals across multiple runs. In the microgrid task, JODC often failed to converge to a positive return, while DEPS consistently found high-performing configurations.
- Physical Intuition: In the drone task, DEPS successfully identified a design with high translational inertia (to resist wind) but low rotational inertia (to remain maneuverable)—a non-trivial trade-off that matches physical intuition.
- Efficiency: DEPS typically reached optimal performance in fewer iterations than JODC, demonstrating better sample efficiency.
Limitations and Future Work
- Requirement for Differentiability: DEPS requires a differentiable model of the environment. It cannot be used with "black-box" simulators where the underlying equations are unknown or non-differentiable (e.g., simulators involving discrete logic or contact physics).
- Horizon Scaling: The computational cost of automatic differentiation grows with the time horizon ($T$). For very long-term problems, the memory requirements might become prohibitive.
- Future Directions: The authors suggest exploring truncated backpropagation to handle longer horizons and combining DEPS with model-learning techniques. If the environment model is unknown, an agent could first learn a differentiable approximation of the physics from data and then apply DEPS to that learned model.
Summary updated