Initial Stabilization in Data Driven Control

181 concepts124 papers1 proposal

Imagine trying to learn how to balance a bicycle on a tightrope. If you fall instantly, you never get the chance to learn from your mistakes. This is the core challenge of data-driven control for open-loop unstable systems: the initial stabilization problem.

In model-free reinforcement learning (RL) and adaptive control, we want to optimize a policy directly from data. However, a fundamental paradox emerges: to safely collect trajectories, we need a stable policy; but to design a stable policy, we need to know or sample the system's dynamics. Priyabrata Saha et al. describe this as a circular dependency in Neural Identification for Control. If an autonomous system starts with zero prior knowledge, random exploration can cause the system states to explode exponentially before any meaningful learning can occur.

The Warm-Start Assumption

Many foundational theoretical guarantees in control theory bypass this problem by assuming it is already solved. For instance, in their seminal work on the global convergence of policy gradient methods for the Linear Quadratic Regulator (LQR), Maryam Fazel et al. assume that the initial controller is already stabilizing (Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator).

This "warm-start" requirement is common across several control paradigms:

Without this initial stabilizing configuration, the mathematical guarantees of convergence and optimality break down because the system can exit its safe operating envelope almost instantly.

Why Unstable Systems Explode

To understand why this initialization bottleneck is so severe, consider the difference between stable and unstable physical systems. In an open-loop stable system, injecting random noise for exploration results in bounded, noisy trajectories. In an open-loop unstable system, the system dynamics contain eigenvalues with positive real parts, meaning any perturbation grows exponentially over time ($e^{At}$).

As Lenart Treven et al. demonstrate, the primary challenge in autonomous control is not finding the optimal cost, but surviving this initial stabilization phase (Learning Stabilizing Controllers for Unstable Linear Quadratic Regulators from a Single Trajectory). Once a stabilizing controller is identified, the system's state growth transitions from exponential to linear, which drastically simplifies subsequent optimization.

If data collection takes too long, the system crashes. Julian Berberich et al. highlight that for systems with high open-loop instability, the initial data collection phase poses a catastrophic failure risk before the controller has acquired enough historical data to generate a valid control law (Linear tracking MPC for nonlinear systems Part II: The data-driven case).

Unstable System (No Initial Policy):  [Exploration Noise] ---> Exponential State Growth ---> Catastrophic Failure
Stable System / Pre-stabilized:       [Exploration Noise] ---> Bounded Trajectories   ---> Successful Learning

Breaking the Circular Dependency

To make data-driven control truly autonomous, researchers have developed mechanisms to stabilize unknown systems from scratch without a pre-existing stable policy.

1. Horizon and Discount Annealing

One way to handle an unstable system is to artificially shorten how far into the future the controller looks. Juan C. Perdomo et al. introduce a model-free algorithm that uses discount annealing (Stabilizing Dynamical Systems via Policy Gradient Methods).

By starting with a very small discount factor (a short effective optimization horizon), the algorithm stabilizes the system over short intervals where exponential growth cannot yet dominate. As the controller learns, the discount factor is incrementally increased. The stability margins of the intermediate controllers tolerate small shifts in dynamics, serving as a safe initialization for the next, more complex optimization task.

2. Unstable Subspace Representation

High-dimensional systems (like fluid flows or power grids) often have complex dynamics but only a few specific modes that are actually unstable. Instead of trying to identify the entire system model from scarce data, we can focus exclusively on the subspace that causes the explosion.

Benjamin Peherstorfer et al. show that by focusing on the small subspace of unstable dynamics, complex systems can be stabilized with minimal data samples (Context-aware controller inference for stabilizing dynamical systems from scarce data).

Expanding on this, Leonardo Felipe Toso proves that learning a low-dimensional representation of the left unstable subspace reduces the sample complexity of finding a stabilizing policy (Learning Stabilizing Policies via an Unstable Subspace Representation). Instead of scaling with the full state dimension squared, the sample complexity scales only with the number of unstable modes squared, making cold-start stabilization practical for high-dimensional systems.

3. Value Function Initialization

In the domain of Adaptive Dynamic Programming (ADP), the initialization bottleneck can also be bypassed by changing how the learning algorithm is initialized. Derong Liu et al. note that by initializing value functions with positive semidefinite functions rather than arbitrary weights, convergence to optimal control can be achieved without requiring a pre-existing stable policy (State of the Art of Adaptive Dynamic Programming and Reinforcement Learning). This mathematical choice ensures that the updates remain bounded and stable throughout the online learning process.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 128 papers
  2. Version 1 · 2026-10-04 · 169 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_initial_stabilizing_policy_requirement_350edd2a,
  title = {initial stabilizing policy requirement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initial-stabilizing-policy-requirement/rdr_350edd2adf394dd1a0d4a60b7f0cb857},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_initial_stabilizing_policy_requirement_350edd2a_v2,
  title = {initial stabilizing policy requirement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initial-stabilizing-policy-requirement/rdr_350edd2adf394dd1a0d4a60b7f0cb857/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake