When building mathematical models of dynamic systems—whether tracking a robot arm, forecasting a time series, or tuning an optimization algorithm—we face a fundamental trap. Standard optimization techniques like gradient descent or Expectation-Maximization evaluate a model based on its short-term, one-step-ahead prediction error. However, when we deploy these models for long-term simulations or recursive forecasting, small errors compound exponentially. If the optimization process inadvertently selects a model with unstable eigenvalues, the simulation quickly explodes.
Traditionally, engineers fixed this post-hoc by checking the model's stability after training and applying corrections, such as clipping eigenvalues or using rejection sampling. Unfortunately, these fixes often destroy the model's accuracy or are computationally prohibitive.
A powerful alternative paradigm has emerged: stability by design. Instead of fixing an unstable model after optimization, we can alter the geometry of the parameter search space itself so that every single point the optimizer can visit is mathematically guaranteed to be stable.
Direct Parametrization: Shaping a Safe Search Space
The most elegant way to guarantee stability is to construct a coordinate system where instability cannot even be expressed. If the unconstrained parameters of our optimization algorithm automatically map to a stable system matrix, the optimizer can run completely unhindered.
This insight dates back to classical time-series analysis. As shown by Benczur et al., the parameter space for autoregressive models is geometrically complex and prone to instability during numerical optimization. By mapping these coefficients to a simpler domain—such as partial correlations—they maintained mathematical stationarity and regularity throughout the estimation process.
Modern machine learning scales this concept to high-dimensional state-space models (SSMs) using specialized matrix structures. In their SIMBa framework, Di Natale et al. ensure linear system stability by utilizing free parametrizations of Schur matrices. Because Schur matrices inherently possess eigenvalues within the unit circle, the underlying system matrix remains stable at every step of backpropagation, eliminating the need for external corrections.
Unconstrained Parameters (ℝⁿ) ───[ Schur / Positive-Definite Mapping ]───► Strictly Stable Manifold ───► Guaranteed Bounded Simulation
For continuous-time stochastic systems, Leth et al. achieve a similar guarantee by defining system matrices as functions of unconstrained free parameters, such as combinations of positive-definite and skew-symmetric matrices. This structural approach satisfies the Stochastic Bounded-Real Lemma by design, ensuring that every parameter configuration represents a physically plausible system and avoiding expensive rejection sampling during Bayesian inference.
By restricting the search space to a manifold of stable matrices, algorithms can maintain iterative stability. As demonstrated by Mamakoukas et al., this allows an optimization algorithm to be safely stopped at any arbitrary point while still providing a mathematically valid and stable dynamical model.
This geometric restriction is equally potent when tuning controllers. Marco et al. restrict optimization to a stable manifold by parameterizing the search space directly through the weights of a Linear Quadratic Regulator (LQR). To handle even higher-dimensional controller spaces, Sarmadi et al. train a Variational Autoencoder (VAE) on a buffer of known stable configurations. The VAE compresses the complex parameter space into a low-dimensional latent representation, forcing subsequent Bayesian optimization to focus exclusively on regions likely to yield stable system behavior.
Embedding Stability via Linear Matrix Inequalities
When a direct structural parametrization is difficult to formulate, we can instead carve out a safe region within the parameter space using Lyapunov stability theory. Lyapunov theory states that a system is stable if there exists an energy-like function that strictly decreases over time. This requirement can often be expressed as a Linear Matrix Inequality (LMI) and solved via Semidefinite Programming (SDP).
Tobenkin et al. pioneered this by integrating dissipation inequalities and Lyapunov-based constraints directly into the optimization objective for identifying nonlinear state-space models. This prevents the accumulation of small prediction errors that typically cause standard equation-error models to diverge over long horizons.
This approach resolves a notorious vulnerability in the Expectation-Maximization (EM) algorithm. Standard EM algorithms often wander into unstable parameter regimes during intermediate iterations. To solve this, Umenberger et al. embedded stability certificates directly into LMI constraints during the optimization process, restricting the search space to stable parameter regimes at every single iteration.
Unconstrained Search Space ───► [ LMI / Lyapunov Filter ] ───► Approved Stable Update
This strategy bridges classical control and modern deep learning frameworks:
- Subspace Identification: Di Natale et al. construct state-transition matrices using an LMI-based parametrization, ensuring eigenvalues remain within the unit circle during optimization without degrading final model accuracy.
- Uncertainty Learning: Ghanipoor et al. formulate the learning of missing system uncertainties as an SDP incorporating Lyapunov-based stability criteria, preventing the erratic behavior common in unconstrained black-box models.
- Deep Koopman Operators: Tian et al. incorporate norm-based stability constraints directly into the training loop of a deep Koopman framework via a differentiable Quadratic Programming (QP) layer, bounding cumulative prediction errors during long-term recursive forecasting.
Projections and Convergence Penalties
When strict structural constraints or LMIs are too computationally expensive to evaluate at every step, optimization frameworks can enforce stability through iterative projections or soft mathematical penalties.
In time-series modeling, Liu et al. introduce explicit projection steps into a proximal optimization framework for fitting ARMA models. These projections mathematically force the estimated parameters back into the interior of the unit circle, guaranteeing the stationarity and invertibility required to prevent divergent forecasts.
Gradient Step ───► [ Temporary Instability ] ───► Projection Step ───► Back inside Unit Circle
This methodology is highly effective in the "Learning to Optimize" (L2O) domain, where a meta-algorithm learns the hyperparameters (like step-sizes) of a discrete optimization algorithm. If the meta-algorithm chooses overly aggressive hyperparameters, the learned algorithm will diverge on high-curvature landscapes.
To bridge continuous-time theory and discrete execution, Yin et al. impose explicit mathematical stability conditions as penalty terms within a stochastic meta-learning framework, ensuring the resulting discrete algorithms remain stable.
From a probabilistic perspective, Sucker et al. implement convergence-constrained priors within a PAC-Bayesian framework. This mechanism restricts the hyperparameter distribution to configurations meeting a user-defined probability of convergence. By acting as a safety rail, it allows the system to aggressively pursue optimization speed while strictly bounding the risk of algorithmic failure.
Key Foundational Work
Convex optimization in identification of stable non-linear state space models (2010) — contributed 2 concept elements to this direction.
Statistical problems of the elementary Gaussian processes : Part 2. : Statistics and related problems (1975) — contributed 1 concept elements to this direction.