Data-driven optimal control of unknown nonlinear dynamical systems using the Koopman operator

Zhexuan Zeng, Ruikun Zhou, Yiming Meng, Jun Liu

Abstract

Nonlinear optimal control is vital for numerous applications but remains challenging for unknown systems due to the difficulties in accurately modelling dynamics and handling computational demands, particularly in high-dimensional settings. This work develops a theoretically certifiable framework that integrates a modified Koopman operator approach with model-based reinforcement learning to address these challenges. By relaxing the requirements on observable functions, our method incorporates nonlinear terms involving both states and control inputs, significantly enhancing system identification accuracy. Moreover, by leveraging the power of neural networks to solve partial differential equations (PDEs), our approach is able to achieving stabilizing control for high-dimensional dynamical systems, up to 9-dimensional. The learned value function and control laws are proven to converge to those of the true system at each iteration. Additionally, the accumulated cost of the learned control closely approximates that of the true system, with errors ranging from $10^{-5}$ to $10^{-3}$.

Introduction

In robotics and aerospace engineering, the goal is often to find the "optimal" trajectory for a system—minimizing fuel consumption, time, or tracking error. This typically requires solving the Hamilton-Jacobi-Bellman (HJB) equation. However, two major hurdles persist: the lack of perfect mathematical models (unknown dynamics) and the "curse of dimensionality," which makes solving the HJB equation computationally prohibitive for high-dimensional systems.

Existing model-free reinforcement learning methods can handle high dimensions but often lack theoretical guarantees and require massive datasets. Conversely, traditional Koopman operator methods, which linearize nonlinear systems in a higher-dimensional space, often struggle to capture the interaction between control inputs ($u$) and states ($x$), leading to poor performance in complex, control-affine scenarios.

Methodology

This research proposes a theoretically certifiable framework that bridges data-driven identification and rigorous control theory through three primary pillars:

1. Modified Koopman Lifting

Standard Koopman methods often treat inputs as separate from states. This framework introduces coupling terms (e.g., products of states and inputs) into the "lifting" functions. This allows the model to accurately recover the control-affine structure ($\dot{x} = f(x) + g(x)u$) essential for physical systems. As shown in the experimental parameters, the domain for identification includes both state and control variables.

Parameters for system identification and Koopman lifting across the four benchmark systems.

2. Resolvent-Based Identification

Instead of numerically unstable methods like taking the logarithm of the Koopman operator, the authors utilize the Yosida approximation. This approach identifies the system's "generator" directly from data, bypassing the strict requirement for perfectly invariant function spaces and making the identification process more robust.

3. Neural Policy Iteration

To overcome the curse of dimensionality, the framework employs Random Feature Neural Networks to approximate the value function. By training only the output layer, the task of solving a complex Partial Differential Equation (PDE) is transformed into an efficient linear least-squares problem. This allows the system to scale to high-dimensional states while maintaining theoretical convergence.

Configuration of the Random Feature Neural Networks used for value function approximation.

Experimental Setup and Results

The framework was evaluated on four benchmark nonlinear systems: a 2D Inverted Pendulum, a 4D Cartpole, a 6D 2D Quadrotor, and a 9D 3D Quadrotor.

Key findings include:

  • Enhanced Accuracy: By including $x \cdot u$ terms in the basis functions, identification errors for the control matrix ($g(x)$) dropped by one to two orders of magnitude compared to standard Koopman baselines.
  • Scalability: The method successfully stabilized a 9-dimensional 3D quadrotor. In contrast, baseline methods failed to stabilize even the 4D cartpole.
  • Near-Optimality: The accumulated cost of the learned controller closely matched the "true" optimal controller (which requires full knowledge of the system equations), with errors ranging from $10^{-5}$ to $10^{-3}$.
  • Theoretical Rigor: The authors provided mathematical proof that the learned value functions and control laws converge to the true system's values at each iteration.

Conclusion

This work demonstrates that combining modified Koopman lifting with neural-network-based policy iteration provides a powerful, theoretically grounded approach to controlling unknown nonlinear systems. While the current implementation relies on polynomial basis functions and ensures local stability, future work will focus on broadening the stabilized region and increasing resilience to noisy real-world sensor data.

Summary updated

Abstract

Nonlinear optimal control is vital for numerous applications but remains challenging for unknown systems due to the difficulties in accurately modelling dynamics and handling computational demands, particularly in high-dimensional settings. This work develops a theoretically certifiable framework that integrates a modified Koopman operator approach with model-based reinforcement learning to address these challenges. By relaxing the requirements on observable functions, our method incorporates nonlinear terms involving both states and control inputs, significantly enhancing system identification accuracy. Moreover, by leveraging the power of neural networks to solve partial differential equations (PDEs), our approach is able to achieving stabilizing control for high-dimensional dynamical systems, up to 9-dimensional. The learned value function and control laws are proven to converge to those of the true system at each iteration. Additionally, the accumulated cost of the learned control closely approximates that of the true system, with errors ranging from $10^{-5}$ to $10^{-3}$.
BibTeX
@misc{zeng2024data,
  author = {Zhexuan Zeng and Ruikun Zhou and Yiming Meng and Jun Liu},
  title = {{Data-driven optimal control of unknown nonlinear dynamical systems using the Koopman operator}},
  year = {2024},
  eprint = {2412.01085},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.01085},
}
Figure