Online convex optimization for data-driven control of dynamical systems

Marko Nonhoff, Matthias A. Müller

Abstract

We propose a data-driven online convex optimization algorithm for controlling dynamical systems. In particular, the control scheme makes use of an initially measured input-output trajectory and behavioral systems theory which enable it to handle unknown discrete-time linear time-invariant systems as well as a priori unknown time-varying cost functions. Further, only output feedback instead of full state measurements is required for the proposed approach. Analysis of the closed loop's performance reveals that the algorithm achieves sublinear regret if the variation of the cost functions is sublinear. The effectiveness of the proposed algorithm, even in the case of noisy measurements, is illustrated by a simulation example.

Introduction

In classical control theory, designing an effective controller typically requires a precise mathematical model of the system, often represented by state-space matrices ($A, B, C, D$). However, modern complex systems like power grids or robotics frequently defy accurate modeling, and their performance objectives—defined by cost functions—may change unpredictably over time.

This paper introduces a data-driven framework for controlling unknown linear systems using only output feedback and pre-recorded data. By integrating Online Convex Optimization (OCO) with behavioral systems theory, the authors develop an algorithm capable of achieving "sublinear regret." This means the controller learns to perform nearly as well as an optimal controller with perfect foresight, even in the presence of time-varying and unknown cost functions. Crucially, this approach addresses the "black box" scenario where the system model is unknown and internal states are not directly measurable.

Methods

The proposed methodology leverages the Fundamental Lemma from behavioral systems theory. This lemma establishes that a single, sufficiently "complex" (persistently exciting) trajectory of a linear system contains all the necessary information to reconstruct any other possible trajectory.

The algorithm avoids explicit system identification by using Hankel Matrices—structured arrangements of raw input-output data—to represent system dynamics. The core control law, Algorithm 1 (Data-Driven Output Feedback), employs Online Gradient Descent (OGD) to track a moving target (the optimal equilibrium). It predicts future outputs over a horizon $\mu$ and solves a constrained optimization problem to determine the optimal control input. A significant theoretical contribution is the data-based steady-state characterization, which defines all possible equilibrium points using raw data alone, bypassing the need for a traditional model.

Results

The algorithm was evaluated using a simulated unstable Linear Time-Invariant (LTI) system with 5 states, 2 inputs, and 1 output. Performance was tested under three conditions: perfect (noiseless) data, noisy pre-recorded data, and noisy online measurements.

As shown in the simulation results, the controller successfully tracks randomly moving optimal targets (indicated by the dashed red lines). The algorithm demonstrates significant robustness; even with noise levels up to $10^{-2}$, it remains stable and continues to track the optimal equilibrium, although performance naturally shows slight degradation compared to the noiseless case.

Theoretically, the authors prove (Theorem 2) that the algorithm achieves sublinear dynamic regret. This implies that the average error per time step converges to zero as time progresses, provided the variation in the environment is sublinear. Furthermore, the control law can be efficiently solved using a weighted pseudoinverse, facilitating real-time application.

Discussion and Limitations

While the proposed framework offers a powerful model-free approach to optimal control, several limitations remain. The current theory is strictly applicable to Linear Time-Invariant (LTI) systems; extending these results to nonlinear systems, where the Fundamental Lemma does not directly apply, is a key area for future research. Additionally, the cost functions must be strongly convex and smooth. Handling non-convex objectives or "economic" cost functions remains an open challenge. Finally, while simulations demonstrate noise tolerance, the formal regret bound proof currently assumes noiseless data, suggesting a need for a rigorous theoretical treatment of noisy environments.

Summary updated

Abstract

We propose an algorithm based on online convex optimization for controlling discrete-time linear dynamical systems. The algorithm is data-driven, i.e., does not require a model of the system, and is able to handle a priori unknown and time-varying cost functions. To this end, we make use of a single persistently exciting input-output sequence of the system and results from behavioral systems theory which enable it to handle unknown linear time-invariant systems. Moreover, we consider noisy output feedback instead of full state measurements and allow general economic cost functions. Our analysis of the closed loop reveals that the algorithm is able to achieve sublinear regret, where the measurement noise only adds an additional constant term to the regret upper bound. In order to do so, we derive a data-driven characterization of the steady-state manifold of an unknown system. Moreover, our algorithm is able to asymptotically exactly estimate the measurement noise. The effectiveness and applicational aspects of the proposed method are illustrated by means of a detailed simulation example in thermal control.
BibTeX
@article{nonhoff2022online,
  author = {Marko Nonhoff and Matthias A. Müller},
  title = {{Online convex optimization for data-driven control of dynamical systems}},
  journal = {IEEE Open Journal of Control Systems},
  year = {2022},
  doi = {10.1109/ojcsys.2022.3200021},
}
Figure