Disturbance Action Parameterization for Nonstochastic Control

78 concepts56 papers1 proposal

In classical control theory, optimizing a policy to stabilize a system and minimize costs is straightforward when the environment is predictable or governed by well-behaved, zero-mean stochastic noise. However, when a system encounters non-stochastic, time-varying, or actively adversarial disturbances, traditional control paradigms struggle.

If you try to optimize a standard state-feedback controller online using gradient descent, you quickly run into a fundamental mathematical wall: the optimization landscape is highly non-convex.

The breakthrough that enables modern online non-stochastic control is a reparameterization technique known as disturbance-action parameterization (or disturbance-response control). By shifting the controller's input from the current state to a history of past environmental shocks, we flatten a notoriously difficult non-convex optimization landscape into a perfectly convex one. This reduction allows us to solve complex, adversarial control problems using standard, efficient tools from Online Convex Optimization (OCO).

The Non-Convexity Trap of State Feedback

To understand why this reparameterization is necessary, consider a standard linear dynamical system:

$$x_{t+1} = Ax_t + Bu_t + w_t$$

where $x_t$ is the state, $u_t$ is the control input, and $w_t$ is an arbitrary disturbance injected by nature at time $t$.

The textbook approach is to use a linear state-feedback policy, $u_t = K x_t$, where we optimize the gain matrix $K$. If we substitute this policy back into the dynamics, the closed-loop system evolves as:

$$x_{t+1} = (A + BK)x_t + w_t$$

If we unroll this equation from the beginning of time, the state $x_t$ becomes a function of past disturbances scaled by powers of the transition matrix:

$$x_t = \sum_{i=0}^{t-1} (A + BK)^i w_{t-1-i}$$

Herein lies the trap. The policy parameters we want to optimize ($K$) are trapped inside an exponentiation. Even if our cost function $c_t(x_t, u_t)$ is perfectly convex with respect to the state and action, it becomes highly non-convex when viewed as a function of $K$. Attempting to update $K$ online via gradient descent can lead to local minima, numerical instability, or catastrophic failure if an update accidentally pushes $A + BK$ into an unstable regime.

The Core Insight: Feedback on Shocks, Not States

The solution to this problem is to change what the controller listens to. Instead of reacting directly to the state $x_t$, the controller reacts to the sequence of past disturbances $w_t$.

To do this safely, we first choose a stable, nominal feedback controller $K_0$ that ensures the baseline matrix $A + BK_0$ is stable. We then parameterize our actual control input as an adjustment to this stable baseline:

$$u_t = K_0 x_t + \sum_{i=1}^{H} M_i w_{t-i}$$

where $M = (M_1, M_2, \dots, M_H)$ are the parameters we want to optimize online, and $H$ is a finite memory horizon.

Because we know the system matrices $A$ and $B$, we can perfectly reconstruct the exact disturbance nature injected at the previous step by observing the state transition:

$$w_t = x_{t+1} - Ax_t - Bu_t$$

Why does this change everything? Crucially, the disturbances $w_t$ are chosen by the environment; they do not depend on our adaptive parameters $M$. When we unroll the dynamics of the system under this new policy, both the state $x_t$ and the control input $u_t$ become affine (linear) functions of the parameters $M$.

As Eugenio Cinquemani et al. demonstrated, parameterizing control actions as affine functions of past noise ensures that the relationship between your optimization variables and the system trajectory remains affine. Because the composition of a convex cost function with an affine mapping is always convex, the entire control problem becomes convex with respect to $M$.

Reducing Control to Optimization with Memory

By mapping $M$ to a convex landscape, we reduce the problem of controlling a physical system to an Online Convex Optimization (OCO) problem. As Elad Hazan et al. and Naman Agarwal et al. formalized, this reduction allows us to deploy standard gradient-based algorithms to minimize regret—the difference between the cumulative cost incurred by our online controller and the best stable linear controller chosen in hindsight.

However, control problems introduce a unique twist that standard online learning does not face: memory. An action taken at time $t$ affects the state $x_{t+1}$, which affects the cost at time $t+1$.

Disturbance-action parameterization elegantly handles this. Because the nominal controller $K_0$ is stable, the physical system acts as a decaying filter. The impact of an action taken in the past vanishes exponentially fast. This allows us to truncate the policy's memory to a small finite horizon $H$.

The author of Making Non-Stochastic Control (Almost) as Easy as Stochastic showed that this bounded memory allows online gradient descent to reliably find stable, optimal policies even when disturbances are malicious. To achieve the absolute limits of performance, Dylan J Foster et al. introduced the Riccatitron algorithm, proving that this convex reparameterization can even yield logarithmic regret bounds in adversarial environments.

Extending the Paradigm

The elegance of reducing control to convex optimization has allowed researchers to extend the disturbance-action framework far beyond simple linear systems.

Safety and Constraints

In real-world applications, controllers must respect physical boundaries (e.g., voltage limits or obstacle avoidance). Because disturbance-action policies keep the state and action trajectories affine in $M$, safety constraints remain convex. Subhro Das et al. and Nanfei Jiang et al. leveraged this by introducing "buffer zones" into the online optimization process. By slightly shrinking the constraint boundaries, they proved that standard gradient updates can maintain absolute safety guarantees while minimizing regret under adversarial noise. Xin Liu et al. further extended this to handle constraints that are themselves time-varying and adversarial.

Partial Observability

When the full state $x_t$ cannot be directly measured, we cannot perfectly calculate the past disturbances $w_t$. To bypass this, Max Simchowitz et al. showed that controllers can instead be parameterized using reconstructed past disturbances derived from noise-free nominal observers. Similarly, Sahin Lale et al. and Gautam Goel utilized classical Youla-like parameterizations to linearize the search space directly over measurement histories, bringing the same convex optimization guarantees to partially observable linear dynamical systems (LDS).

Model-Free Reinforcement Learning and Deep Control

What if the underlying physics ($A$ and $B$) are completely unknown? Udaya Ghai et al. bypassed this by introducing pseudo-disturbances—proxy signals calculated directly from data that substitute for true environmental shocks. This allows model-free reinforcement learning algorithms to inherit the stability and convexity of disturbance-response control.

Even when moving to deep learning, where neural networks replace linear filters, the underlying intuition remains powerful. Jason D. Lee et al. demonstrated that by structuring overparameterized neural networks to take historical environmental disturbances as inputs, the networks behave nearly convexly, bridging the gap between deep control heuristics and rigorous online optimization theory.

Key Foundational Work

Convexity and convex approximations of discrete-time stochastic control problems with constraints (2011) — contributed 2 concept elements to this direction.
On convex problems in chance-constrained stochastic model predictive control (2009) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 62 papers
  2. Version 1 · 2026-10-04 · 106 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_control_to_optimization_reduction_3fbb94d7,
  title = {control-to-optimization reduction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/control-to-optimization-reduction/rdr_3fbb94d776ff41e18618e8ae21e521db},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_control_to_optimization_reduction_3fbb94d7_v2,
  title = {control-to-optimization reduction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/control-to-optimization-reduction/rdr_3fbb94d776ff41e18618e8ae21e521db/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake