Classical control theory and standard reinforcement learning algorithms typically rely on a convenient assumption: the underlying physics of a system are stationary. Whether governing a robot's joints, an autonomous vehicle's tire grip, or a power grid's inertia, the system parameters are treated as fixed constants.
In the real world, this assumption breaks down. Mechanical parts wear out, payloads change mid-operation, and environmental conditions shift unpredictably. When the parameters governing a system vary over time, controllers face a fundamental dilemma. If they rely on historical data, their models become obsolete and pull the system toward an outdated reality. If they adapt too aggressively to recent observations, they risk overcorrecting for noise and causing catastrophic instability.
To solve this, modern research focuses on tracking non-stationary system dynamics in real time, balancing the need for rapid adaptation with rigorous guarantees of safety and stability.
The Challenge of a Moving Target
When system parameters drift or jump, the core problem transitions from a static optimization task to a continuous tracking problem. Standard adaptive control laws are designed to handle unknown parameters, but they assume those parameters eventually settle. Under true time-varying uncertainty, the controller must optimize its actions while its internal model of the world is constantly decaying.
To quantify performance in these environments, researchers look at dynamic regret—the difference between the controller's cumulative loss and the loss of the optimal sequence of policies chosen in hindsight with perfect knowledge of the parameter changes. Minimizing dynamic regret requires the controller to continuously estimate the rate of parameter drift and adjust its exploration strategy accordingly.
Tracking Parameter Drift
How can a system figure out how its own physics are changing? Researchers have converged on three primary strategies: discounting the past, event-driven learning, and structural model banks.
Sliding Windows and Memory Management
The simplest way to handle a changing environment is to forget the distant past. By utilizing a sliding window of recent data, a controller can ensure its model is trained only on data that reflects the current state of the world.
For instance, Laurent et al. use a sliding window to continuously re-estimate system dynamics and update control policies via gradient descent, ensuring stability even when mode transitions are completely unknown. When state observations are incomplete, data-driven frameworks like the one developed by Zhai et al. leverage reservoir computing to reconstruct hidden parameter drifts from only a subset of observed variables.
Event-Triggered Learning
Continuously updating a model can be computationally expensive and introduces unnecessary noise when the system is relatively stable. Instead of learning at every time step, event-triggered learning (ETL) uses statistical tests to monitor performance.
As shown by von Rohr et al., the controller only "invests" in a new data-collection and learning phase when a statistical anomaly indicates that the current model has become obsolete due to environmental drift. This prevents the system from destabilizing itself by chasing transient noise.
Model Banks for Abrupt Shifts
When parameters change discontinuously—such as a race car suddenly hitting a patch of wet track—sliding windows adapt too slowly. To handle these instantaneous jumps, AL-Sunni et al. utilize a "look-back" window paired with a massive predefined bank of physics models. Rather than gradually converging on new parameters, the system evaluates recent history against the bank and instantly switches to the best-fitting model.
Continuous Drift ---> [Sliding Window / Reservoir Computing] ---> Smooth Adaptation
Abrupt Jumps ---> [Look-Back Window + Model Bank] ---> Instantaneous Switching
Guaranteeing Stability and Performance
Tracking parameters is only half the battle; the controller must still execute safe actions while its estimates are shifting. This requires algorithmic mechanisms that explicitly account for parameter movement.
Dynamic Regret Bounds
To prove that an algorithm can handle non-stationarity, researchers bound its dynamic regret relative to the total variation of the system ($V_T$). Luo et al. achieved this with Dyn-LQR, an algorithm that pairs adaptive non-stationarity detection with certainty-equivalent control to guarantee tight regret bounds in linear quadratic regulator systems. For adversarial settings where targets move unpredictably, Abbasi-Yadkori et al. demonstrated that controllers can achieve low regret by treating the shifting system matrices as an online optimization problem.
Managing Exploration Uncertainty
In dual control frameworks, the controller must actively explore the environment to learn parameters while simultaneously maintaining control. This creates a secondary problem: as the controller learns, its average estimate of the system parameters shifts, which can induce instability.
Venkatasubramanian et al. solved this by applying gain scheduling, treating future uncertain parameter estimates as scheduling variables. This mathematical framework ensures that the system remains robustly stable even as the controller's internal beliefs are actively shifting during the exploration phase.
Local vs. Global Accuracy
A common bottleneck in controlling complex, non-affine, or nonlinear systems is the impossibility of learning a perfect global model on the fly. However, Preiss et al. proved that global accuracy is unnecessary. By introducing the M-GAPS framework, the authors demonstrated that a controller can maintain low local regret as long as it accurately predicts the dynamics along the trajectory it actually visits, provided the underlying environment varies slowly.
Real-World Implications
Accounting for time-varying parameters is essential for moving advanced control out of the lab and into production:
- Aerospace and Robotics: Drones and quadrotors experience massive structural changes during flight as fuel or liquid payloads are consumed. Hu et al. showed that embedding these changing moments of inertia directly into the control state allows for seamless adaptation. Furthermore, Evangelisti notes that assuming a standard autonomous Lagrangian system fails when a robot interacts with external loads, requiring data-driven models that explicitly allow parameters to evolve.
- Power Grid Infrastructure: As renewable energy sources replace traditional spinning generators, power grids lose their physical buffer of inertia. Feng et al. addressed this non-stationary volatility by designing an online switching framework that dynamically selects the best-tuned controller as grid inertia fluctuates.
Ultimately, the shift from static parameter estimation to time-varying tracking allows control systems to operate safely in volatile environments, transforming non-stationarity from a source of failure into a manageable parameter.
Key Foundational Work
Training and Tracking in Robotics (1985) — contributed 1 concept elements to this direction.
Formulation of a ^|^ldquo;Multiple Targets Problem^|^rdquo; and Its Analysis (1970) — contributed 1 concept elements to this direction.