Spectral Recovery Controllers: Fixing Imitation Drift Without Expert Queries

A policy trained by copying demonstrations, called behavior cloning, works until it makes a small mistake. Once it drifts into states the demonstrator never visited, it has no idea how to get back, and small errors snowball. Interactive imitation methods fix this by asking an expert to label the states the learner actually visits (DAgger), by perturbing the expert during data collection so recoveries get recorded (DART), or by teleporting the learner back onto the demonstrated path in a simulator. All three either cost expert time or need resets that physical systems cannot perform. A fourth limit is often overlooked. Every one of these methods copies the demonstrator's own corrections, so a policy can never handle disturbances better than the person who demonstrated.

This note proposes a different fix. Keep the cloned policy as the nominal controller. Add a small residual controller that watches how far the system has drifted from the demonstrated trajectory and learns online, from that drift alone, how to push it back. The residual controller uses fixed spectral filtering. The recent history of disturbances is compressed by convolving it with a handful of precomputed filters, the top eigenvectors of a universal Hankel matrix, a matrix built from a system's impulse response. Choosing the correction then becomes a convex online learning problem with regret guarantees. The bet is that drift around a demonstration behaves like a nearly linear system pushed by structured, long-memory disturbances, which is the regime where spectral control is provably efficient. If that holds, recovery needs no expert queries and no resets. The policy can also reject disturbances better than the demonstrator did, because it competes with the best disturbance-feedback controller in hindsight rather than with the demonstrator.

Core Claim

Compounding error in imitation is mostly a disturbance-rejection failure in the drift dynamics. Online spectral control can learn a recovery controller for those dynamics using only the observed deviation from the reference, with three properties. Expert labels are not needed after the demonstrations are collected. Rollouts start from the system's natural initial states, with no resets. Per-step compute grows only polylogarithmically with the system's memory length.

The policy is

$$u_t = \pi_{BC}(o_t) + \sum_{k=1}^{K} M_k \sum_{i=1}^{L} \phi_k(i)\, \hat w_{t-i},$$

where $u_t$ is the applied action and $\pi_{BC}$ is the cloned policy acting on observation $o_t$. The term $\hat w_t$ is the estimated disturbance in the deviation dynamics: the part of the next deviation from the demonstrated reference that a local linear model of those dynamics does not explain. The $\phi_k$ are the top $K$ eigenvectors of the fixed $L \times L$ Hankel matrix $Z_{ij} = (1-\gamma)^{i+j-1}/(i+j-1)$, which depends only on a stability-margin lower bound $\gamma$. The $M_k$ are action-space matrices updated by projected online gradient descent on a deviation-plus-action cost. Because the cost is convex in the $M_k$, the learning problem stays convex even when the history length $L$ is long.

The "beyond the demonstrator" part of the claim is deliberately narrow. The residual cannot discover new strategies, such as a different gait or a route the demonstrator never took. It can only reject disturbances better than the demonstrator did. Value-based stitching methods target a different kind of ceiling, and the experiments keep the two apart.

Where This Sits

The method comes from online spectral control. Learning Linear Dynamical Systems via Spectral Filtering showed that the Hankel spectrum of any stable linear system decays exponentially, so a few fixed filters suffice for prediction. A New Approach to Controlling Linear Dynamical Systems turned this into Online Spectral Control (OSC). OSC matches gradient predictive control's loss with 15 spectral features where the earlier method needed 50 raw disturbance features, and its runtime is polylogarithmic in $1/\gamma$. Efficient Spectral Control of Partially Observed Linear Dynamical Systems extended this to partial observations through a "naturalizer," which strips out the effect of the controller's own past actions, and reports beating LQG under sinusoidal disturbances. Both papers assume known system matrices and never touch demonstrations. We reuse their controller and replace "known dynamics" with "local deviation dynamics estimated around a demonstration."

Flash STU shows that fixed spectral filters model MuJoCo dynamics well in open loop: validation MSE 0.0092 on Ant-v1, against 0.0139 for Mamba-2 and 0.0237 for Transformers. It never closes the loop, so it says nothing about drift. This motivates the representation but does not test our question.

Two lines of imitation work motivate the gap. DART avoids DAgger's supervisor burden by injecting noise during demonstration. It reports 79% grasp success versus 49% for behavior cloning, and it notes that DAgger's training rollouts on Humanoid earn 80% less reward than the supervisor. DART still needs the expert in the loop during collection, and its recoveries are only as good as the expert's. Non-Adversarial Imitation Learning Provably Free of Compounding Errors proposes Dual Q-DM. It achieves linear rather than quadratic error growth by letting Bellman constraints carry value from expert states into unvisited ones, but it needs a tuned neural critic with a penalty coefficient. It also shows that IQ-Learn collapses toward behavior cloning, which constrains us: a recovery signal must have a direction, and uniformly suppressing values at unvisited states is not enough.

Two further papers constrain the design. Inverse Reinforcement Learning without Reinforcement Learning relies on expert resets, which we rule out. Provably Breaking the Quadratic Error Compounding Barrier shows that knowing the dynamics helps in tabular problems, giving $\Theta(H^{1.5})$ error. We test a continuous, partially observed analogue in which an estimated local linear model stands in for known transitions.

Research Program

The first component is estimating the drift dynamics. Around each demonstration we fit a local linear model of how the deviation evolves under action perturbations. Locomotion is periodic, so the model is indexed by gait phase. The fit is ordinary least squares on spectral features, using the demonstrations plus a small budget of dithered rollouts of the cloned policy. Residuals of this model define $\hat w_t$.

One weakness is unavoidable: model error leaks into $\hat w_t$ and makes it depend on the controller's own actions. That breaks the "oblivious disturbance" assumption behind OSC's regret bound. We measure how large this leakage is. We also test the naturalizer-style correction from Double Spectral Control (DSC), which helps when only partial observations are available.

The second component is the residual controller. It runs OSC, or the double spectral version when observations are partial, with phase-binned matrices $M_k$. Per-bin matrices remain linear in the parameters, so the update stays convex. The comparator class contains every stabilizing linear disturbance-feedback controller around the reference.

The third component is a modest theoretical result. Suppose the cloned policy's per-step error $\epsilon$ enters the drift dynamics as a bounded disturbance, and the comparator class contains a stabilizing controller. Then the regret bound should imply cumulative deviation cost of order $\epsilon H + \sqrt{H}$, rather than behavior cloning's $\epsilon H^2$. The $\sqrt{H}$ term is OSC's regret rate; the linear term comes from the comparator absorbing each error within a bounded time. We will state the conditions, including a model-error term, explicitly rather than claim generality.

Baselines

The naive reference is plain behavior cloning on the same demonstrations, using the same network as $\pi_{BC}$. Every other method is reported relative to it and to the demonstrator's own return under identical disturbances.

The strongest competitor that avoids expert queries is Dual Q-DM. It is the only method in the file that provably removes quadratic compounding without adversarial training, and it uses environment rollouts much as we do. The file reports no numeric MuJoCo returns for it; it states only that Dual Q-DM beats IQ-Learn and LS-IQ in demonstration efficiency. We will run it from its description on the IQ-Learn and LS-IQ codebases, with the same demonstrations, the same rollout budget, and an equal hyperparameter sweep that includes its penalty coefficient. LS-IQ runs alongside it as a second value-based reference.

The strongest expert-in-the-loop competitor is DART. The comparison number is its 79% grasp success against 49% for behavior cloning. We give DART its noise-injected demonstrations at a matched number of expert transitions. DART is allowed expert time and our method is not, so ties favor us, but we report the comparison either way.

Three ablations isolate the key ingredient. The first keeps the same residual structure but uses a time-varying LQR on the estimated model; it tests whether learning online beats fixed linear feedback. The second replaces the spectral history with a raw disturbance history of length $L$, which is gradient predictive control. The third swaps spectral features for a small learned recurrent encoder trained by the same online gradient updates.

Beating these baselines means four things on held-out disturbance types. The method must reach higher return than Dual Q-DM at equal rollout budget. It must exceed the demonstrator's return under structured disturbances. It must show a fitted error-versus-horizon exponent near 1, where behavior cloning should sit near 2. And it must beat the LQR ablation on long-memory disturbances.

Why not just…?

Why not just use DART? DART is a real threat when disturbances are memoryless noise that the expert handles well, and it may tie or win there. It fails in two places by construction. It needs the expert during collection, and its labels inherit the expert's own recovery skill, so it cannot beat a demonstrator who handles slow sinusoidal drift poorly. The disturbance-memory sweep below settles where each method wins.

Why not just bolt a tracking LQR onto behavior cloning? This is the most serious alternative, and it may suffice for Gaussian disturbances. LQR is optimal for white noise but not for structured or adversarial disturbances; DSC reports beating LQG under sinusoidal disturbances, though the file gives no numbers. The LQR ablation, run on the identical estimated model, directly measures how much online spectral adaptation adds. If the gap vanishes across disturbance types, we will report that the contribution is the residual-tracking structure, not spectral filtering.

Why not just train a history-aware sequence policy, such as Flash STU, Mamba, or a Transformer, by behavior cloning? A longer context improves open-loop prediction, as Flash STU's Ant-v1 results show. It does not create training data off the demonstrated path, so covariate shift remains. We include a Flash STU policy trained by behavior cloning and expect it to beat plain cloning at short horizons but still show quadratic error growth.

Why not just use a value-based method like Dual Q-DM? It is the principled alternative. Its costs are a tuned neural critic, and a recovery signal that propagates only as far as rollouts have explored. Our advantage, if any, should appear in sample efficiency and in tuning sensitivity. We report performance across its penalty sweep, not just its best setting.

Proposed Experiments

The first test bed is the synthetic linear systems from OSC and DSC: state dimension 10, with Gaussian, Rademacher, sinusoidal, and ReLU-nonlinear disturbances. Demonstrations come from a deliberately mistuned LQR. Ground truth is known here, so we can measure regret, model-error leakage, and the horizon exponent exactly. We also vary $\gamma$ to confirm that runtime stays polylogarithmic.

The second test bed is MuJoCo Hopper, Walker2d, HalfCheetah, and Ant. Demonstrations come from the iMuJoCo pretrained SAC checkpoints, including medium-quality ones. We inject "tremble" noise as in Swamy et al., plus structured disturbances: periodic pushes, slowly drifting actuator bias, and sinusoidal ground forces. A partially observed variant masks velocities to exercise the naturalizer.

Metrics are return normalized to the demonstrator under the same disturbance, the fitted exponent of cumulative deviation versus horizon $H$, rollouts to convergence, the count of expert queries and resets (zero for our method), and per-step compute.

Three checks are decisive. The method must show error growth near linear in $H$ while behavior cloning shows near quadratic growth, on both test beds. It must exceed the demonstrator under sinusoidal and bias disturbances while matching it under white noise; if it also exceeds the demonstrator under white noise, that suggests the gain comes from something other than disturbance rejection and needs explaining. And its margin over the LQR ablation must grow with disturbance memory.

A negative result would also be informative. If model-error leakage into $\hat w_t$ dominates on contact-rich tasks such as Hopper, that bounds where linear spectral recovery applies, and it points to the unresolved need for spectral control with unknown dynamics.