Choosing an Adaptive Controller's Memory and Step Size Online from Its Own Prediction Errors

When a physical system changes while it runs, the controllers that learn on the fly have to answer a question nobody tunes for them: how much of the past to trust. Tires wear down, a payload is dropped, a motor's load inertia jumps. An adaptive controller keeps re-estimating the system's dynamics from recent data and re-optimizing its feedback law. The two knobs that decide whether it tracks well or falls apart are its memory (a sliding-window length or forgetting factor) and its policy step size. Both are fixed before deployment. Tuning them offline is the usual hyperparameter-optimization problem, with many full runs per configuration, and it gives a single compromise value that is wrong whenever the drift rate changes.

This note argues that in adaptive control the cost of tuning these knobs online can be made nearly free, because the system supplies a clean validation signal at every step. Each new state is a held-out test of every candidate model's one-step prediction. We propose running a small bank of estimators that differ only in memory length. They share the same streaming statistics, so evaluating eight memory settings costs a small constant factor rather than eight runs. A switching-aware expert-weighting rule then picks among them. Policy-side knobs, such as the gradient step size and probing-noise level, are selected by evaluating candidate gains in closed form under the currently selected model, not by reusing logged trajectories. The claim is that this "prequential" hyperparameter selection, meaning scoring each candidate on the next observation before it is used for fitting, beats both fixed tuned hyperparameters and gradient-based online hyperparameter tuning under abrupt parameter jumps. We expect it to cost well under one extra controller's worth of compute.

Core Claim

Hyperparameter tuning for learning-based controllers is expensive only when it is framed as an outer loop. For the memory hyperparameter specifically, every candidate setting can be scored on the same data stream at negligible marginal cost. Candidates whose sufficient statistics nest, such as recursive least squares with a geometric grid of forgetting factors, or sliding windows with nested lengths, update for roughly $K$ times the cost of a single estimator's state-space update. The validation loss is each candidate's one-step prediction error on the incoming state.

Selection uses a fixed-share weighting over the $K$ candidates, a standard online-learning rule that lets weight move quickly back to a previously poor expert. The resulting controller adapts its memory to the drift rate. It does not smooth across discontinuities, because short-memory experts win immediately after a jump and long-memory experts regain weight during quiet periods. The weights are

$$w_{t+1}^{k} \propto (1-\rho)\, w_t^{k} e^{-\beta \ell_t^{k}} + \frac{\rho}{K}\sum_j w_t^{j} e^{-\beta \ell_t^{j}},$$

where $\ell_t^{k}=\|x_{t+1}-\hat A_t^{k}x_t-\hat B_t^{k}u_t\|^2$ is candidate $k$'s prediction error, $\beta$ is a learning rate, and $\rho$ is a switching rate. To avoid replacing one hyperparameter with two, $\rho$ and $\beta$ are themselves placed on a coarse grid and aggregated one level up. Standard fixed-share analysis makes the cost of not knowing them logarithmic in grid size.

The difference from the closest prior work is the source of the tuning signal and its cost. We do not differentiate through the controller, and we do not replay old trajectories. We reuse shared statistics and score candidates on the next observation.

Relation to Prior Work

Three works motivate the gap. Laurent et al.'s policy gradient adaptive control for switched systems re-estimates dynamics by least squares on a sliding window and takes gradient steps on the feedback gain. Its stability proof needs the window to be shorter than the dwell time in each mode, while the window and step size are chosen by hand. That is exactly the knowledge a deployed system lacks. M-GAPS separates online policy optimization from parameter estimation in nonlinear time-varying systems with constant memory, but also leaves its step sizes static. Luo et al.'s Dyn-LQR proves that any fixed sliding window has dynamic regret of at least $\Omega(V_T^{1/3}T^{2/3})$ in linear-quadratic control, where $V_T$ is the total drift of the dynamics. Adaptive restarts driven by multi-scale tests achieve $\tilde O(V_T^{2/5}T^{3/5})$. This lower bound is the strongest argument that memory must adapt. Dyn-LQR is also our most serious competitor, because its restarts are themselves a form of memory adaptation.

Two lines supply the contrasting mechanisms. Online hyperparameter tuning, including hypergradient descent on the learning rate, greedy validation-gradient tuning of regularizers at about 30% overhead, and self-tuning networks, all assume a stationary data distribution. SmoothFBO extends hypergradient tuning to drifting distributions by averaging hypergradients over a time window and proves sublinear local regret. Its authors report catastrophic forgetting in their RL experiment, and its averaging assumes smooth drift. LLA-MPC shows the opposite regime: on racing cars hitting low-friction patches, any averaging lags fatally. Its fix is to score a bank of 20,000 friction models on a 0.2-second look-back window and pick the best. We borrow LLA-MPC's bank-and-score idea but apply it to the learning hyperparameters rather than the physical parameters, so the bank needs no prior grid over physics.

Two event-triggered methods constrain the design. Parameter-filter event-triggered learning shows that updating the model only when a $\chi^2$ test flags a change beats continuous updating. Its Kalman-filter noise covariances and test threshold stay fixed. ET-GP-UCB resets a Bayesian optimizer when observations exit uniform error bounds. Both decide when to relearn, while our mechanism decides how much past to use. A trigger is one point in our expert space (memory reset to zero), and the experiments compare against it directly.

HOOF tunes policy-gradient hyperparameters within one run by importance-weighting already-collected trajectories. Under physical drift those weights mix policy change with dynamics change and degenerate. We replace them with model-based evaluation. For linear plants, the cost of each candidate gain under the currently selected $(\hat A,\hat B)$ is one Lyapunov solve.

Research Program

The work has three parts.

Estimator bank. We will build the memory-bank estimator for linear plants: recursive least squares with $K\in\{4,8,16\}$ forgetting factors on a geometric grid, plus a nested sliding-window variant. We will measure its marginal cost per step against the base estimator.

Policy-knob selection. Each step, the gain update is evaluated for a small grid of step sizes and probing-noise variances under the selected model. The candidate with the lowest predicted closed-loop quadratic cost is executed, and a stability-margin filter rejects any candidate gain that is unstable under any high-weight model. This filter is how we handle the safety tension between exploration and state blow-up.

Analysis. We will attempt a dynamic-regret bound that adds a term for selection error. The conjecture is that it is logarithmic in $K$ and linear in the number of switches, from the fixed-share argument. This bound would sit on top of the window-dependent estimation error. If the bound with the best window in hindsight recovers the adaptive rate of Dyn-LQR up to grid resolution, that answers the question of whether the field's open problem of jointly bounding plant drift and hyperparameter selection is tractable.

For nonlinear plants (CartPole, racing), the bank is a set of locally linear or small learned models fit with different memories. The policy step uses model predictive control with the selected model.

Baselines

The naive reference is the same adaptive controller with hyperparameters fixed at defaults, plus the "permanent updates" estimator from parameter-filter event-triggered learning. On that paper's DC motor benchmark, with load-inertia jumps at $t=1000$ and $t=2000$, it has mean squared parameter error 3.322. Event-triggered learning itself reaches 1.065 in the same paper, which is the strongest reported number we have. Beating it means lower parameter error than 1.065 on the same simulated motor and jump schedule, with zero state or voltage constraint violations. We will reimplement both methods and confirm we reproduce the reported numbers before comparing.

The strongest offline baseline is a single fixed window and step size chosen by BOHB over a training distribution of drift schedules. It gets a tuning budget of 100 full simulated episodes. It is evaluated on held-out schedules whose drift rates fall both inside and outside the training range. Beating it means lower dynamic regret on out-of-range schedules and no worse regret in-range, while our method uses zero tuning episodes.

The strongest theory-backed baseline is Dyn-LQR with its multi-scale restart tests, run on the linear benchmarks with the same exploration-noise budget. Its constants receive the same grid tuning we give ours.

The strongest online-HPO baseline is SmoothFBO-style smoothed hypergradient descent on the forgetting factor and step size, following SmoothFBO, with an unsmoothed hypergradient variant following Baydin et al.. Both receive the same per-step compute as our bank.

On racing, the reference is LLA-MPC, which reports zero warm-up versus 10–60 seconds for APACRace. No reported numerical lap times were found on Lacuna for LLA-MPC, so we will report lap time and spin-outs from our own reimplementation.

Ablations isolate the key ingredient. The first replaces fixed-share with picking the best expert on a fixed look-back window, LLA-MPC style, which tests whether switching-aware weighting matters. The second removes model-based policy-knob evaluation and keeps the memory bank only. The third replaces model-based policy evaluation with HOOF-style importance weighting, which tests our claim that it degenerates under physical drift.

Why not just…?

Why not just use Dyn-LQR's adaptive restarts, which already adapt memory with optimal rates? This is a real threat on linear-quadratic problems. Restarts discard all data at a detected change. Our bank can keep long-memory estimates for components that did not change and can recover gracefully from false alarms. That advantage is plausible but unproven. The 4×4 switched-system and DC motor experiments compare the two head-to-head at equal exploration noise, including sequences of small jumps near the detection threshold. If Dyn-LQR matches us there, our contribution narrows to the policy-knob selection and to nonlinear plants, and we will say so.

Why not just tune a robust fixed configuration offline? The fixed-window lower bound in Dyn-LQR says no fixed memory is rate-optimal when drift varies. The self-tuning networks result shows that retraining with the best fixed values found by an adaptive schedule underperforms the schedule itself. The out-of-range drift test directly measures how much the fixed choice loses and what tuning budget it consumed.

Why not just run hypergradient descent on the forgetting factor? Hypergradients are local. After an abrupt jump the useful forgetting factor is far away, and a step-size-limited gradient walks there slowly. Smoothing, which SmoothFBO needs for variance control, makes the walk slower still, which is the tension LLA-MPC exposes. Our comparison measures recovery time after each jump at matched compute. If smoothed hypergradients win under slow drift, that is a regime boundary worth reporting, and a hybrid would follow.

Why not just use event-triggered relearning? Event-triggered learning already handles jumps well on the DC motor. However, its threshold and filter covariances are fixed, so we expect it to fire falsely or late as noise and drift rates change. We test it on the motor with the noise level varied by a factor of four, with its threshold tuned at the nominal level.

Proposed Experiments

Benchmarks. We will use four testbeds:

  • the DC motor with inertia jumps from event-triggered learning;
  • the 4×4 linear switched system from Laurent et al., extended so that dwell times are drawn from a range spanning shorter and longer than any fixed window;
  • CartPole with the friction jump at $t=100$ from ET-GP-UCB, plus a smooth-drift variant echoing SmoothFBO;
  • the 1:43 racing simulator with tire wear and oil-slick shocks from LLA-MPC.

We will release the drift and switch schedules with controllable total variation and switch counts. This addresses the absence of non-stationary physical tasks in HPOBench.

Metrics. We will report:

  • dynamic regret against a clairvoyant controller;
  • mean squared parameter error;
  • recovery time to within 10% of the post-jump optimal cost;
  • constraint violations;
  • spin-outs and lap time on the racing simulator;
  • per-step compute relative to the base controller, plus total tuning episodes consumed.

Decisive checks. Three outcomes decide the project.

  1. Recovery time after jumps and regret in quiet phases. Our method should match short-window recovery and long-window quiet-phase error simultaneously, which no single fixed window can.
  2. Marginal overhead at $K=8$. It must be below the 30% that greedy regularization tuning pays per step for the "nearly free" claim to stand. Otherwise the claim drops to "cheaper than outer-loop search."
  3. The head-to-head with Dyn-LQR on small-jump sequences, which decides whether shared-statistics memory selection adds anything beyond statistically driven restarts.