Scalable Optimization for Direct Data Driven Predictive Control

37 concepts32 papers1 proposal

In traditional control theory, managing a complex system requires a two-step pipeline: first, perform system identification to build a mathematical model (such as a state-space or differential equation model), and second, use that model to design a controller.

Direct data-driven control turns this paradigm on its head. Frameworks like Data-Enabled Predictive Control (DeePC), introduced by Jeremy Coulson et al. in Data-Enabled Predictive Control: In the Shallows of the DeePC, bypass explicit system identification entirely. Instead of optimizing over an estimated parametric model, these methods embed raw historical data trajectories directly into the real-time optimization loop.

While this approach eliminates model mismatch errors, it introduces a severe computational challenge: the data-dependent optimization scaling bottleneck. Because the data is the model, the size of the optimization problem grows with the volume of collected data, creating a direct conflict between representation accuracy and real-time feasibility.

The Mechanism of the Scaling Bottleneck

To understand why direct data-driven control suffers from a computational explosion, we must look at its mathematical foundation: Willems' Fundamental Lemma.

As reviewed by Frank Allgöwer et al. in An overview of systems-theoretic guarantees in data-driven model predictive control, the lemma states that any valid input-output trajectory of a linear time-invariant system can be expressed as a linear combination of previously recorded trajectories, provided the historical inputs are sufficiently "excited."

In practice, engineers arrange historical input-output data into a large Hankel matrix $H$. To predict and optimize future behavior over a horizon, the controller solves an online optimization problem to find a weight vector $g$ that aligns the historical data with the system's current initial state:

$$\text{Future Trajectory} = H \cdot g$$

Here lies the problem: the dimension of the decision variable $g$ scales linearly with the number of columns in $H$, which corresponds directly to the length of the historical dataset $T$.

As Julian Berberich et al. demonstrate in Data-Driven Model Predictive Control with Stability and Robustness Guarantees, a large $T$ is essential in real-world environments to average out measurement noise and capture complex dynamics. However, as $T$ expands, the number of decision variables in the online Quadratic Program (QP) or Semidefinite Program (SDP) balloons.

This scaling behavior makes direct data-driven control computationally prohibitive for high-frequency applications, such as grid-connected power converters studied by Linbin Huang et al. in Data-Enabled Predictive Control for Grid-Connected Power Converters. Furthermore, in adaptive settings where data is continuously collected online, the authors of Data-Driven Min-Max MPC for Linear Systems: Robustness and Adaptation and Robust MPC for Uncertain Linear Systems - Combining Model Adaptation and Iterative Learning note that the number of constraints grows indefinitely over time, eventually violating strict real-time engineering latency limits.

Traditional MPC:  [State/Horizon Size] ---> [Fixed-Size Optimization] ---> Fast Real-Time Execution
Direct Data-Driven: [Historical Data T] ---> [Growing Weight Vector g] ---> Computational Bottleneck

Decoupling Complexity via Alternative Parameterizations

To resolve this bottleneck, recent research focuses on decoupling the optimization problem's dimensionality from the data volume. The core intuition is to compress the historical data into fixed-size mathematical structures before solving the online optimization, ensuring that data volume only alters the problem's coefficients, not its number of decision variables.

Covariance Parameterization

In Regularization for Covariance Parameterization of Direct Data-Driven LQR Control, the authors introduce a method that shifts the optimization from the high-dimensional weight vector $g$ to the second-order statistical moments of the data. By formulating the control law using sample covariance matrices, the size of the optimization problem becomes a function of the system's input-output channels and the control horizon, remaining completely invariant to whether the covariance was calculated from 1,000 or 1,000,000 data points.

Linear Fractional Transformations (LFT)

For systems where both inputs and outputs are corrupted by noise (the errors-in-variables model), Felix Brändle leverages the Sherman-Morrison-Woodbury formula in A System Parametrization for Direct Data-Driven Analysis and Control with Error-in-Variables. By transforming the data-driven regression into an LFT, the author computes guaranteed robust performance bounds using a single optimization problem that maintains a fixed complexity independent of data volume.

Linear Programming Formulations

For specific system classes, such as positive linear systems (where states represent non-negative quantities like chemical concentrations), the computational burden can be reduced by changing the optimization class itself. In Data-Driven Control of Positive Linear Systems using Linear Programming, the authors apply the Extended Farkas Lemma to map the data-driven robust stabilization problem into a Linear Program (LP). This avoids the heavy computational overhead of Semidefinite Programming (SDP), offering a highly scalable alternative for large-scale systems.

Accelerating the Loop with Learning and Nonconvexity

Another powerful strategy shifts the data-dependent computational burden entirely offline, or restructures the optimization landscape to navigate it more efficiently.

Algorithm Unrolling and Learned Approximations

Instead of solving the full, data-heavy optimization problem at every time step, the authors of Learning-Based Efficient Approximation of Data-enabled Predictive Control replace the online scoring component of DeePC with a learned, fixed-size neural network approximation. By utilizing differentiable convex programming and algorithm unrolling, they truncate an iterative optimization solver into a fixed number of neural network layers. This shifts the data-scaling penalty to an offline training phase, yielding a 5x speedup during real-time execution.

Exploiting Nonconvex Landscapes

While convex relaxations are highly valued for their global optimality guarantees, they often introduce a massive number of auxiliary variables that scale poorly with trajectory length. In Nonconvex Linear System Identification with Minimal State Representation, the authors demonstrate that optimizing directly over a nonconvex, minimal state representation (using Burer-Monteiro factorization) can bypass this dimensional explosion. Surprisingly, the nonconvex landscape remains benign enough to navigate quickly, removing the traditional computational penalty associated with long observation histories.

Managing the Noise-Scalability Trade-off

When operating in noisy environments, data-driven controllers must incorporate robust constraints, which further compounds the optimization complexity.

As proven by Jeremy Coulson et al. in Regularized and Distributionally Robust Data-Enabled Predictive Control, adding standard regularization terms to a data-driven objective function is mathematically equivalent to performing Distributionally Robust Optimization (DRO) against worst-case noise distributions. However, full DRO formulations scale poorly as more historical data samples are incorporated (Qi Li et al. in A Distributionally Robust Model Predictive Control for Static and Dynamic Uncertainties in Smart Grids).

To maintain robust performance without inflating the online optimization problem, the authors of An Extended Kalman Filter for Data-Enabled Predictive Control propose a split architecture:

  1. Offline Data Condensation: An offline averaging technique compresses and "cleans" the large historical data trajectories, reducing the size of the Hankel matrix before it enters the controller.
  2. Online State Filtering: An online Extended Kalman Filter (EKF) leverages the controller's internal optimization structure to filter out real-time measurement noise dynamically.

By separating data volume from online optimization, this architecture allows the controller to exploit large historical datasets for noise resilience while maintaining a lightweight, real-time capable decision variable footprint.

Key Foundational Work

Data-Enabled Predictive Control: In the Shallows of the DeePC (2019) — contributed 2 concept elements to this direction.
Data-Enabled Predictive Control for Grid-Connected Power Converters (2019) — contributed 2 concept elements to this direction.
Data-Driven Model Predictive Control with Stability and Robustness Guarantees (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_data_dependent_optimization_scaling_4eb3f142,
  title = {data-dependent optimization scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-dependent-optimization-scaling/rdr_4eb3f1423d1f4838ab8e26916ae1dc8a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_data_dependent_optimization_scaling_4eb3f142_v1,
  title = {data-dependent optimization scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-dependent-optimization-scaling/rdr_4eb3f1423d1f4838ab8e26916ae1dc8a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake