Abstract
We propose a data-driven online convex optimization algorithm for controlling dynamical systems. In particular, the control scheme makes use of an initially measured input-output trajectory and behavioral systems theory which enable it to handle unknown discrete-time linear time-invariant systems as well as a priori unknown time-varying cost functions. Further, only output feedback instead of full state measurements is required for the proposed approach. Analysis of the closed loop's performance reveals that the algorithm achieves sublinear regret if the variation of the cost functions is sublinear. The effectiveness of the proposed algorithm, even in the case of noisy measurements, is illustrated by a simulation example.
Introduction
In classical control theory, designing an effective controller typically requires a precise mathematical model of the system, often represented by state-space matrices ($A, B, C, D$). However, modern complex systems like power grids or robotics frequently defy accurate modeling, and their performance objectives—defined by cost functions—may change unpredictably over time.
This paper introduces a data-driven framework for controlling unknown linear systems using only output feedback and pre-recorded data. By integrating Online Convex Optimization (OCO) with behavioral systems theory, the authors develop an algorithm capable of achieving "sublinear regret." This means the controller learns to perform nearly as well as an optimal controller with perfect foresight, even in the presence of time-varying and unknown cost functions. Crucially, this approach addresses the "black box" scenario where the system model is unknown and internal states are not directly measurable.
Methods
The proposed methodology leverages the Fundamental Lemma from behavioral systems theory. This lemma establishes that a single, sufficiently "complex" (persistently exciting) trajectory of a linear system contains all the necessary information to reconstruct any other possible trajectory.
The algorithm avoids explicit system identification by using Hankel Matrices—structured arrangements of raw input-output data—to represent system dynamics. The core control law, Algorithm 1 (Data-Driven Output Feedback), employs Online Gradient Descent (OGD) to track a moving target (the optimal equilibrium). It predicts future outputs over a horizon $\mu$ and solves a constrained optimization problem to determine the optimal control input. A significant theoretical contribution is the data-based steady-state characterization, which defines all possible equilibrium points using raw data alone, bypassing the need for a traditional model.
Results
The algorithm was evaluated using a simulated unstable Linear Time-Invariant (LTI) system with 5 states, 2 inputs, and 1 output. Performance was tested under three conditions: perfect (noiseless) data, noisy pre-recorded data, and noisy online measurements.
As shown in the simulation results, the controller successfully tracks randomly moving optimal targets (indicated by the dashed red lines). The algorithm demonstrates significant robustness; even with noise levels up to $10^{-2}$, it remains stable and continues to track the optimal equilibrium, although performance naturally shows slight degradation compared to the noiseless case.
Theoretically, the authors prove (Theorem 2) that the algorithm achieves sublinear dynamic regret. This implies that the average error per time step converges to zero as time progresses, provided the variation in the environment is sublinear. Furthermore, the control law can be efficiently solved using a weighted pseudoinverse, facilitating real-time application.
Discussion and Limitations
While the proposed framework offers a powerful model-free approach to optimal control, several limitations remain. The current theory is strictly applicable to Linear Time-Invariant (LTI) systems; extending these results to nonlinear systems, where the Fundamental Lemma does not directly apply, is a key area for future research. Additionally, the cost functions must be strongly convex and smooth. Handling non-convex objectives or "economic" cost functions remains an open challenge. Finally, while simulations demonstrate noise tolerance, the formal regret bound proof currently assumes noiseless data, suggesting a need for a rigorous theoretical treatment of noisy environments.
Summary updated