Regret Optimal Control for Unknown Dynamical Systems

This direction is merged into another direction: generalization to nonlinear systems. Its last version stays readable and citable here.

Version 1 of 1164 papers169 conceptsfrozen 2026-10-04

In the traditional divide of autonomous systems, two schools of thought have long operated in parallel. Control theory, the bedrock of aviation and robotics, prioritizes stability—ensuring a system doesn't "blow up" and eventually reaches its target. Reinforcement Learning (RL), the engine of modern AI, prioritizes regret—minimizing the cumulative cost incurred while learning how to act.

As Benjamin Recht argues in A Tour of Reinforcement Learning: The View from Continuous Control, these fields are increasingly merging into a single pursuit: designing controllers that can navigate unknown, complex environments with the sample efficiency of RL and the rigorous safety guarantees of control theory.

The Benchmark of Hindsight

The central mechanism for bridging these fields is dynamic regret. Unlike standard optimization, which compares an agent to a fixed best action, dynamic regret compares an agent’s performance to a "clairvoyant" optimal policy—one that knew the entire sequence of future disturbances and system changes in advance.

Gautam Goel has shown in Regret-optimal control in dynamic environments that minimizing this regret allows a controller to automatically adapt to its environment. If the disturbances are random noise, the controller acts like a Kalman filter; if the disturbances are strategic attacks, it acts like a robust $H_\infty$ controller. By reducing the regret-minimization problem to the classical Nehari problem, researchers have found ways to build adaptive controllers that compete with the best possible sequence of actions chosen in hindsight, even under partial observability.

From Linear Foundations to Nonlinear Frontiers

Most of our theoretical "gold standards" come from the Linear Quadratic Regulator (LQR). In this setting, we assume the world is linear and the costs are quadratic. Hesameddin Mohammadi and Mihailo R. Jovanovic proved in Convergence and Sample Complexity of Gradient Methods for the Model-Free Linear–Quadratic Regulator Problem that even model-free RL can achieve global convergence in these systems.

However, the real world is rarely linear. The transition to nonlinear dynamics introduces a "stability gap." As Nicholas M. Boffi and Jean-Jacques E. Slotine explain in Regret Bounds for Adaptive Nonlinear Control, early estimation errors in nonlinear systems can compound exponentially, leading to catastrophic failure before the agent has a chance to learn. While linear systems can achieve "logarithmic" regret (where learning slows down quickly), nonlinear systems often struggle with "square-root" regret, representing a fundamental performance floor that researchers are still trying to break.

Controlling Without a Model

One of the most promising directions for handling this complexity is Data-Enabled Predictive Control (DeePC). Instead of trying to identify a mathematical model of the system (which might be wrong), DeePC uses raw historical data to predict future behavior directly.

Jeremy Coulson, John Lygeros, and Florian Dorfler have provided the rigorous foundation for this in Regularized and Distributionally Robust Data-Enabled Predictive Control. They proved that the common practice of adding "regularization" to these controllers is mathematically equivalent to Distributionally Robust Optimization (DRO). In other words, by penalizing complex solutions, the controller is implicitly protecting itself against the worst-case noise distribution in the data.

The Cost of Learning

A recurring theme across this research is the trade-off between exploration and exploitation. To minimize regret, an agent must explore the system to learn its dynamics. But in physical systems, exploration is dangerous.

Algorithms like StabL, introduced by Sahin Lale in Reinforcement Learning with Fast Stabilization in Linear Dynamical Systems, address this by using a strategic "burst" of exploration to stabilize the system first, before attempting to optimize it. Similarly, James A. Preiss and Yiheng Lin suggest in Online Policy Optimization in Unknown Nonlinear Systems that we don't need a perfect global model of the world. We only need to be accurate along the trajectory the agent is actually visiting.

Why It Works

The intuition behind the success of these modern regret-optimal frameworks is robustness through adaptation. By framing control as an online learning problem, we move away from "one-size-fits-all" controllers. Instead, we create agents that:

  1. Quantify Uncertainty: Using tools like Thompson Sampling or scenario optimization (Andrea Martin et al., Regret optimal control for uncertain stochastic systems).
  2. Minimize Regret: Competing against the best possible hindsight policy to ensure they aren't just stable, but efficient.
  3. Leverage Structure: Using kernel methods or symmetry (Hamed Taghavian and Mikael Johansson 0001) to reduce the amount of data needed to reach optimal performance.

The ultimate goal, as seen in the work of Andreas Krause on NeoRL, is to enable agents to learn in the "real world"—where there is no reset button, and every mistake has a lasting cost.

Papers in this version

Showing 60 of 164 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_regret_optimal_control_for_unknown_dynam_29295438,
  title = {Regret Optimal Control for Unknown Dynamical Systems},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/regret-optimal-control-for-unknown-dynamical-systems/rdr_292954389add4807b2567e3a32a36e1f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_regret_optimal_control_for_unknown_dynam_29295438_v1,
  title = {Regret Optimal Control for Unknown Dynamical Systems},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/regret-optimal-control-for-unknown-dynamical-systems/rdr_292954389add4807b2567e3a32a36e1f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}