Jointly Learning Environments and Control Policies with Projected Stochastic Gradient Ascent

Adrien Bolland, Ioannis Boukas, Mathias Berger, Damien Ernst

TL;DR

This paper introduces DEPS, a novel deep reinforcement learning algorithm that jointly learns both the dynamics of an environment and a control policy for it. This approach, which leverages projected stochastic gradient ascent, significantly outperforms existing methods in achieving higher expected returns with fewer iterations across several complex simulated environments.

What Problem Does This Paper Address?

Many real-world engineering problems involve designing a system and then controlling it. Traditionally, these two aspects—design and control—are optimized separately. However, the authors argue that this separation can lead to suboptimal overall performance because the optimal design and the optimal control policy are often interdependent. For instance, the best way to control a robot hand might depend on its physical design, and vice-versa. This paper tackles the challenge of jointly optimizing both the system's design parameters (environment parameters) and its control policy simultaneously to achieve superior performance. The core difficulty lies in the high-dimensional and potentially non-convex nature of this joint optimization problem.

Methods

The paper proposes a novel algorithm called Direct Environment and Policy Search (DEPS). DEPS combines two key ideas:

  • Policy Gradient Methods: These are standard in reinforcement learning for learning control policies. DEPS extends this by considering the gradients with respect to both policy parameters and environment parameters.
  • Model-Based Optimization: DEPS leverages the analytical form of the environment's transition and reward functions (assumed to be known and differentiable) to compute gradients. This is combined with Monte-Carlo sampling and automatic differentiation to estimate these gradients.

The algorithm iteratively updates both the environment parameters ($\psi$) and the policy parameters ($\theta$) using projected gradient ascent. This projection step is crucial for ensuring that the updated parameters remain within valid, constrained ranges. The core idea is to treat the entire problem as a single optimization task in the joint parameter space of the environment and the policy.

Datasets

The paper does not use traditional datasets for training. Instead, it introduces and evaluates its algorithm on three newly designed simulated environments:

  • Mass-Spring-Damper (MSD) Environment: A classic control problem adapted for joint design and control.
  • Off-Grid Microgrid Environment: A complex system for designing and operating a power grid to minimize costs.
  • Drone Environment: A quadrotor control problem focused on achieving an elliptical trajectory quickly while subject to wind disturbances.

These environments are designed to be challenging and representative of real-world joint design and control problems.

Key Insights and Ideas

  • Joint Optimization is Crucial: The paper strongly demonstrates that jointly optimizing the environment design and the control policy leads to significantly better performance (higher expected returns) compared to optimizing them separately or using rule-based policies.
  • DEPS Offers Stability and Efficiency: DEPS consistently converges faster and exhibits lower variance in its results compared to the state-of-the-art JODC algorithm, suggesting it is more stable and sample-efficient.
  • Leveraging Analytical Models: By assuming differentiable environment dynamics, DEPS can compute more accurate gradients, leading to more effective optimization.
  • Projected Gradient Ascent for Constraints: The use of projected gradient ascent is essential for handling the practical constraints on system design parameters, ensuring that the learned designs are physically realizable.

Key Results

The DEPS algorithm was evaluated across the three introduced environments and consistently outperformed the JODC algorithm (a state-of-the-art method for joint design and control) and rule-based optimization approaches.

  • Mass-Spring-Damper: DEPS achieved an average expected return of 99.90, very close to the theoretical maximum, with minimal variance. It converged much faster than JODC.
  • Off-Grid Microgrid: DEPS achieved a positive average expected return of 47.60, while JODC resulted in a negative return (-24.17).
  • Drone: DEPS achieved a final expected return of 23.20, outperforming JODC's 16.39. The learned drone design from DEPS also resulted in a more efficient trajectory following.

In all cases, DEPS yielded solutions with higher returns in fewer iterations.

Limitations and Future Work

  • Smoothness Assumption: DEPS relies on the assumption that the environment's transition and reward functions are differentiable. This limits its applicability to systems with non-smooth dynamics.
  • Computational Complexity for Long Horizons: The computational cost of automatic differentiation scales with the length of the optimization horizon, potentially making DEPS intractable for very long-term problems.
  • Future Work:
    • Investigate methods to handle non-smooth environments.
  • Develop techniques to mitigate the computational cost for long horizons, possibly by truncating backpropagation.
  • Combine DEPS with recent advancements in actor-critic methods and other direct policy search techniques for improved learning stability and exploration.
  • Explore learning approximate representations of environment dynamics instead of assuming they are known, bridging DEPS with model-based reinforcement learning.

Summary updated

Abstract

We consider the joint design and control of discrete-time stochastic dynamical systems over a finite time horizon. We formulate the problem as a multi-step optimization problem under uncertainty seeking to identify a system design and a control policy that jointly maximize the expected sum of rewards collected over the time horizon considered. The transition function, the reward function and the policy are all parametrized, assumed known and differentiable with respect to their parameters. We then introduce a deep reinforcement learning algorithm combining policy gradient methods with model-based optimization techniques to solve this problem. In essence, our algorithm iteratively approximates the gradient of the expected return via Monte-Carlo sampling and automatic differentiation and takes projected gradient ascent steps in the space of environment and policy parameters. This algorithm is referred to as Direct Environment and Policy Search (DEPS). We assess the performance of our algorithm in three environments concerned with the design and control of a mass-spring-damper system, a small-scale off-grid power system and a drone, respectively. In addition, our algorithm is benchmarked against a state-of-the-art deep reinforcement learning algorithm used to tackle joint design and control problems. We show that DEPS performs at least as well or better in all three environments, consistently yielding solutions with higher returns in fewer iterations. Finally, solutions produced by our algorithm are also compared with solutions produced by an algorithm that does not jointly optimize environment and policy parameters, highlighting the fact that higher returns can be achieved when joint optimization is performed.
BibTeX
@article{bolland2022jointly,
  author = {Adrien Bolland and Ioannis Boukas and Mathias Berger and Damien Ernst},
  title = {{Jointly Learning Environments and Control Policies with Projected Stochastic Gradient Ascent}},
  journal = {JAIR},
  year = {2022},
  doi = {10.1613/jair.1.13350},
}
Figure