Learning Over All Stabilizing Nonlinear Controllers for a Partially-Observed Linear System

Ruigang Wang, Nicholas Barbara, Max Revay, Ian R. ManchesterIEEE Control. Syst. Lett 2023 · OpenReview

TL;DR

This paper introduces a novel nonlinear controller architecture for partially-observed linear systems that guarantees closed-loop stability during learning. By combining the Youla parameterization with Recurrent Equilibrium Networks (RENs), they enable unconstrained optimization for learning stabilizing controllers, outperforming existing methods in simulations.

What Problem Does This Paper Address?

Deep reinforcement learning (RL) has shown great promise in control tasks, but its application in real-world engineering is hindered by a lack of stability guarantees, especially in scenarios where not all system states are directly observable. Existing methods for partially-observed systems often require complex, non-convex optimization techniques to maintain stability. This paper tackles the challenge of learning stabilizing controllers for such systems while ensuring that the system remains stable throughout the learning process, even when dealing with unobserved states.

Methods

The core of the proposed solution lies in a nonlinear extension of the classical Youla parameterization. Traditionally, the Youla parameterization allows for the representation of all stabilizing controllers for linear systems using a linear "Q" parameter. This paper extends this to nonlinear systems by using a nonlinear operator, specifically a Recurrent Equilibrium Network (REN), as the Q parameter.

Here's a breakdown of the key methodological components:

  • Youla Parameterization: The authors leverage the Youla parameterization to create a structure where any stabilizing controller can be represented by a stable linear system combined with a nonlinear Q-parameter. This structure inherently ensures stability if the Q-parameter itself is stable and contracting.
  • Recurrent Equilibrium Networks (RENs): RENs are a class of dynamic neural networks that have been shown to be universal approximators of contracting and Lipschitz nonlinear systems. By using a REN as the Q-parameter, the authors gain a flexible and powerful tool for learning nonlinear control policies.
  • Unconstrained Optimization: A significant advantage of the REN architecture is that it allows for unconstrained optimization. This is a major improvement over prior work that required complex projection methods to enforce stability constraints.
  • Two Learning Scenarios: The paper considers two learning settings:
  • Model-Based Case: Utilizes exact gradients from a differentiable simulation model.
  • Reinforcement Learning (Random Search): Employs zeroth-order oracles (only cost function evaluations) and uses a random search method to approximate gradients, making it suitable for experimental settings.

Datasets

Not discussed in the paper. The experiments are conducted using a simulated linearized cart-pole system.

Key Insights and Ideas

  • Universal Approximator for Stabilizing Controllers: The paper proves that the proposed Youla-REN architecture is a universal approximator for stabilizing nonlinear controllers for partially-observed linear systems. This means, in theory, it can represent any such controller.
  • Stability Guarantees During Learning: The most crucial insight is that the REN architecture, when used within the Youla parameterization, inherently guarantees closed-loop stability throughout the learning process. This is a significant step towards deploying learned controllers in safety-critical applications.
  • Unconstrained Optimization Simplifies Learning: By moving away from constrained optimization (like SDPs) to unconstrained optimization, the learning process becomes significantly simpler and more scalable.
  • RENs as Flexible Building Blocks: The paper highlights RENs as powerful, flexible building blocks for learning dynamic systems with guaranteed properties, extending their applicability beyond just system identification to controller design.

Key Results

  • Improved Performance: In simulation experiments on a linearized cart-pole system, the Youla-REN controller consistently outperformed other methods (including feedback policies and a projection-based RNN controller) in terms of reaching lower final costs.
  • Faster Convergence: The Youla-REN controller converged faster than other Youla-based approaches (e.g., Youla-LSTM) and achieved better final performance.
  • Scalability and Efficiency: The proposed method was orders of magnitude faster than the projection-based method [6] in terms of computation time per epoch, demonstrating its scalability.
  • Robustness to Learning Method: The Youla-REN controller performed well in both model-based (exact gradients) and reinforcement learning (random search) settings, showcasing its adaptability.

Limitations and Future Work

  • Theoretical Focus: The paper primarily focuses on linear systems with nonlinear controllers. Extending this to fully nonlinear systems is a natural next step.
  • Simulation-Based Validation: While simulations show promising results, real-world experimental validation is needed to confirm the practical efficacy and robustness of the approach.
  • Complexity of RENs: While RENs offer advantages, their internal complexity and the number of parameters can still be substantial, potentially impacting sample efficiency in very complex scenarios.
  • Future Work: The authors suggest exploring the application of this framework to more complex systems, investigating different types of nonlinear operators beyond RENs, and further analyzing the theoretical properties of the learned controllers.

Summary updated

Abstract

This letter proposes a nonlinear policy architecture for control of partially-observed linear dynamical systems providing built-in closed-loop stability guarantees. The policy is based on a nonlinear version of the Youla parameterization, and augments a known stabilizing linear controller with a nonlinear operator from a recently developed class of dynamic neural network models called the recurrent equilibrium network (REN). We prove that RENs are universal approximators of contracting and Lipschitz nonlinear systems, and subsequently show that the proposed Youla-REN architecture is a universal approximator of stabilizing nonlinear controllers. The REN architecture simplifies learning since unconstrained optimization can be applied, and we consider both a model-based case where exact gradients are available and reinforcement learning using random search with zeroth-order oracles. In simulation examples our method converges faster to better controllers and is more scalable than existing methods, while guaranteeing stability during learning transients.
BibTeX
@inproceedings{wang2023learning,
  author = {Ruigang Wang and Nicholas Barbara and Max Revay and Ian R. Manchester},
  title = {{Learning Over All Stabilizing Nonlinear Controllers for a Partially-Observed Linear System}},
  booktitle = {IEEE Control. Syst. Lett},
  year = {2023},
  url = {https://openreview.net/forum?id=0lV2fSb5YT},
}
Figure