In classical control theory, we often assume that the "noise" hitting a system is well-behaved—usually Gaussian white noise. This assumption allows us to derive elegant solutions like the Linear Quadratic Regulator (LQR). However, real-world systems face "nonstochastic" disturbances: wind gusts that don't follow a distribution, adversarial attacks, or structural changes like mechanical wear.
Elad Hazan, Sham M. Kakade, and Karan Singh formalized this in The Nonstochastic Control Problem, shifting the goal from minimizing expected cost to minimizing regret. Regret measures how much worse our controller performed compared to the best possible controller we could have chosen if we had known all the future disturbances in advance.
The Convexity Trap
The fundamental challenge in online control is that the relationship between a standard controller's parameters and the resulting cost is usually non-convex.
Consider a simple linear system: $x_{t+1} = Ax_t + Bu_t + w_t$, where $w_t$ is the disturbance. If we use a standard state-feedback law $u_t = Kx_t$, the state $x_t$ becomes a product of the matrix $(A+BK)$ over time. This creates a high-degree polynomial relationship between the gain $K$ and the cost. In optimization terms, this landscape is "bumpy" and full of local minima, making it nearly impossible to use efficient tools like Gradient Descent to update the controller in real-time.
The Solution: Disturbance-Response Parameterization
To fix this, researchers introduced a clever change of variables known as disturbance-response (or disturbance-action) parameterization.
Instead of deciding the action $u_t$ based on the current state $x_t$, the controller maintains a memory of past disturbances $\{w_{t-1}, w_{t-2}, \dots, w_{t-H}\}$ and chooses an action that is a linear combination of them: $$u_t = \sum_{i=1}^{H} M_i w_{t-i}$$ As Naman Agarwal and colleagues demonstrate in Logarithmic Regret for Online Control, this shift is transformative. Because the system is linear, the state $x_t$ and the cost become convex functions of the parameters $M_i$.
By parameterizing the policy this way, we turn a "bumpy" control problem into a "bowl-shaped" optimization problem. This allows us to apply Online Convex Optimization (OCO), where we can iteratively improve our controller using simple gradient updates as new disturbances are observed.
Why It Works: Canceling the Past
The intuition behind this approach is that disturbances are the "shocks" that push a system away from its desired path. If a controller can see the history of these shocks, it can learn a "filter" (the $M$ matrices) that proactively cancels out their lingering effects.
This mechanism is powerful enough to handle:
- Adversarial Noise: Since the optimization doesn't care about the distribution of $w_t$, it works even if an adversary is choosing the noise to hurt the system.
- Unknown Dynamics: In Black-Box Control for Linear Dynamical Systems, Xinyi Chen and Elad Hazan show that even if the system matrices $A$ and $B$ are unknown, we can estimate the disturbances and still achieve sublinear regret.
- Hard Constraints: Subhro Das and Na Li showed in Online Optimal Control with Affine Constraints that this convex structure allows for "buffer zones" that guarantee safety even under unpredictable noise.
Achieving Optimal Performance
The field has rapidly progressed toward finding the most efficient algorithms for this framework. Dylan J. Foster and Max Simchowitz introduced Riccatitron in Logarithmic Regret for Adversarial Online Control, proving that for certain systems, the regret only grows logarithmically with time—the gold standard for "fast learning."
Similarly, Asaf Cassel and Tomer Koren extended these ideas to the "bandit" setting in Bandit Linear Control, where the controller only sees the cost it incurred, not the full state, yet can still learn the optimal disturbance-response filter.
Beyond Linear Models
While the theory is rooted in linear systems, the "disturbance-based" intuition is spreading. In Online Nonstochastic Model-Free Reinforcement Learning, Udaya Ghai and others use "pseudo-disturbances" to make model-free RL robust to adversarial changes. Even deep learning is being integrated; Jason D. Lee and Edgar Minasyan have shown in Regret Guarantees for Online Deep Control that overparameterized neural networks can act as these disturbance-response filters, bringing the theoretical rigor of nonstochastic control to the flexibility of deep learning.