Jointly Learning Environments and Control Policies with Projected Stochastic Gradient Ascent

Adrien Bolland, Ioannis Boukas, Mathias Berger, Damien Ernst

TL;DR

DEPS (Direct Environment and Policy Search) is a new deep reinforcement learning algorithm that jointly optimizes a system's physical design and its control policy. By leveraging differentiable models and projected gradient ascent, DEPS achieves higher performance with better sample efficiency and stability than existing state-of-the-art methods across diverse engineering benchmarks.

What Problem Does This Paper Address?

In traditional engineering, design and control are often treated as sequential steps: first, you build the hardware (the "environment"), then you write the software to control it (the "policy"). However, these two aspects are deeply intertwined. For example, a drone's arm length affects both its structural stability and its maneuverability. Designing them separately often leads to sub-optimal performance.

The challenge lies in the complexity of the joint optimization space. The objective function—maximizing expected rewards over time—is typically non-convex, high-dimensional, and subject to uncertainty. Previous attempts to solve this used either derivative-free methods (which are slow and scale poorly) or complex reinforcement learning (RL) approaches that optimize a distribution over designs, which can be noisy and unstable.

Methods

The authors introduce Direct Environment and Policy Search (DEPS). The core idea is to treat the physical design parameters ($\psi$) and the control policy parameters ($\theta$) as a single vector to be optimized simultaneously.

Key technical components include:

  • Differentiable Programming: DEPS assumes that the system's transition function (physics) and reward function are known and differentiable. This allows the use of automatic differentiation (autodiff) to compute exact gradients.
  • Projected Stochastic Gradient Ascent: The algorithm uses Monte-Carlo sampling to estimate the gradient of the expected return. It then takes a step in the direction of the gradient and "projects" the result back into the feasible parameter space (e.g., ensuring a battery capacity isn't negative).
  • Direct Parameter Search: Unlike previous methods like JODC that learn a probability distribution over designs, DEPS optimizes the design parameters directly. This significantly reduces variance and improves convergence speed.

Datasets

The authors evaluated DEPS on three distinct, increasingly complex simulated environments:

  1. Mass-Spring-Damper (MSD): A classic physics benchmark where the goal is to stabilize a mass at a specific position by optimizing spring constants and damping coefficients alongside the controller.
  2. Off-Grid Microgrid: A socio-technical system where the agent must size a battery, solar panels, and a diesel generator to minimize the total cost of energy over a lifetime.
  3. Drone Design: A high-dimensional robotics task where the agent optimizes the arm length, thickness, and propeller radius of a quadrotor to fly an elliptical path as fast as possible despite wind gusts.

The implementation and benchmarks are available on GitHub.

Key Insights and Ideas

  • The "Direct" Advantage: By optimizing design parameters directly rather than learning a distribution over them, DEPS avoids the high variance that plagues distributional methods. This makes the learning process much more stable.
  • Hardware as a Differentiable Layer: The paper treats the physical environment as just another layer in a neural network. If you can differentiate through the physics, you can use the same tools (like Adam or SGD) used to train AI to design hardware.
  • Co-dependency is Crucial: The experiments show that a policy optimized for one design performs poorly if the design changes slightly. Joint optimization finds a "sweet spot" where the hardware's physical properties actually make the control task easier.
  • Sample Efficiency: Because DEPS uses the analytical form of the environment to compute gradients, it requires significantly fewer simulations to reach an optimal solution compared to model-free or distributional RL methods.

Key Results

  • Superior Performance: In all three environments, DEPS achieved higher expected returns than the JODC (Joint Optimization of Design and Control) baseline.
  • Stability: DEPS showed much tighter confidence intervals across multiple runs. In the microgrid task, JODC often failed to converge to a positive return, while DEPS consistently found high-performing configurations.
  • Physical Intuition: In the drone task, DEPS successfully identified a design with high translational inertia (to resist wind) but low rotational inertia (to remain maneuverable)—a non-trivial trade-off that matches physical intuition.
  • Efficiency: DEPS typically reached optimal performance in fewer iterations than JODC, demonstrating better sample efficiency.

Limitations and Future Work

  • Requirement for Differentiability: DEPS requires a differentiable model of the environment. It cannot be used with "black-box" simulators where the underlying equations are unknown or non-differentiable (e.g., simulators involving discrete logic or contact physics).
  • Horizon Scaling: The computational cost of automatic differentiation grows with the time horizon ($T$). For very long-term problems, the memory requirements might become prohibitive.
  • Future Directions: The authors suggest exploring truncated backpropagation to handle longer horizons and combining DEPS with model-learning techniques. If the environment model is unknown, an agent could first learn a differentiable approximation of the physics from data and then apply DEPS to that learned model.

Summary updated

Abstract

We consider the joint design and control of discrete-time stochastic dynamical systems over a finite time horizon. We formulate the problem as a multi-step optimization problem under uncertainty seeking to identify a system design and a control policy that jointly maximize the expected sum of rewards collected over the time horizon considered. The transition function, the reward function and the policy are all parametrized, assumed known and differentiable with respect to their parameters. We then introduce a deep reinforcement learning algorithm combining policy gradient methods with model-based optimization techniques to solve this problem. In essence, our algorithm iteratively approximates the gradient of the expected return via Monte-Carlo sampling and automatic differentiation and takes projected gradient ascent steps in the space of environment and policy parameters. This algorithm is referred to as Direct Environment and Policy Search (DEPS). We assess the performance of our algorithm in three environments concerned with the design and control of a mass-spring-damper system, a small-scale off-grid power system and a drone, respectively. In addition, our algorithm is benchmarked against a state-of-the-art deep reinforcement learning algorithm used to tackle joint design and control problems. We show that DEPS performs at least as well or better in all three environments, consistently yielding solutions with higher returns in fewer iterations. Finally, solutions produced by our algorithm are also compared with solutions produced by an algorithm that does not jointly optimize environment and policy parameters, highlighting the fact that higher returns can be achieved when joint optimization is performed.
BibTeX
@article{bolland2022jointly,
  author = {Adrien Bolland and Ioannis Boukas and Mathias Berger and Damien Ernst},
  title = {{Jointly Learning Environments and Control Policies with Projected Stochastic Gradient Ascent}},
  journal = {JAIR},
  year = {2022},
  doi = {10.1613/jair.1.13350},
}
Figure