Engineers still set controller parameters by hand: the gains of a feedback loop, or the weights that tell a model predictive controller how much to trade tracking accuracy against actuator effort. A model predictive controller (MPC) is a controller that, at every time step, solves a short optimization problem over a predicted future trajectory. Two automated alternatives exist, and each has a limitation. Gradient methods differentiate through the controller's internal optimization problem. They are fast, but they rely on the controller's internal dynamics model, so the gradients are biased whenever that model is wrong. Model-free methods perturb the parameters and measure the real cost. They are unbiased, but very noisy, and they need many real trials. A family of variance-reduction methods for stochastic optimization, called recursive momentum, makes noisy gradients cheap to average. These methods carry a running gradient estimate forward and correct it by how much the gradient changed between consecutive parameter settings. Their core trick has a catch for robotics: it evaluates the same random sample at the old and the new parameters. On real hardware the wind gust, the road friction and the sensor noise of a trial cannot be replayed.
This note proposes a way around that catch. The real system supplies the fresh, unbiased but noisy gradient term. The correction term, meaning how much the gradient changed from the previous parameters to the current ones, is computed by replaying the disturbances measured on that real trial through the differentiable model at both parameter settings. The model's error enters only through a difference of gradients at nearby parameters. That error shrinks with the step size instead of persisting as a fixed bias. The expected result is an automated tuner that needs a few noisy real trials per step, keeps the low variance of model-based gradients, and still converges toward the true system's optimum rather than the model's.
Core Claim
Recursive momentum (STORM, Momentum-Based Variance Reduction in Non-Convex SGD; SARAH, SARAH) keeps an estimate $G_t$ of the gradient of the true closed-loop cost $J$ with respect to controller parameters $\theta$. We propose the update
$$G_t = \alpha\, g_t + (1-\alpha)\big(G_{t-1} + \nabla \hat J(\theta_t;\hat\xi_t) - \nabla \hat J(\theta_{t-1};\hat\xi_t)\big).$$
Here $g_t$ is an unbiased but noisy gradient estimate from one real rollout at $\theta_t$, obtained by randomized parameter perturbation as in Zheng et al. The term $\alpha\in(0,1]$ controls how strongly the estimate is re-anchored to real data. $\hat J(\theta;\hat\xi)$ is the closed-loop cost of the differentiable MPC run on its nominal model with an additive disturbance sequence $\hat\xi$. The sequence $\hat\xi_t$ is reconstructed from the real trajectory at step $t$ as one-step residuals, real next state minus model-predicted next state. The two model gradients share one disturbance realization, so most of their noise cancels. This is the "same sample twice" property, recovered in a setting where the real sample cannot be rerun.
The new ingredient is the error analysis. Write $\delta_t$ for the mismatch between the model-replayed gradient change and the true gradient change. If the real and model gradients are Lipschitz with constants differing by at most $\Delta L$, then $\|\delta_t\| \le \Delta L\,\|\theta_t-\theta_{t-1}\|$. Unrolling the recursion bounds the accumulated bias by $\Delta L\,\eta\, \|G\|/\alpha$, where $\eta$ is the step size. The model's error is therefore scaled down by the step size. That is the opposite of purely model-based differentiable MPC, whose bias is fixed by the model error no matter how small the steps. The claim to test is that this estimator reaches a given closed-loop cost in several times fewer real rollouts than the gray-box hybrid, and stays unbiased where model-only gradients are not.
Where this sits in prior work
Three papers supply the controller-side machinery. Differentiable MPC for End-to-end Planning and Control differentiates MPC solutions through their optimality conditions, the equations that hold at the solver's optimum. Differentiable Model Predictive Control on the GPU makes this fast by parallelizing the solve across time steps. We reuse both to compute $\nabla\hat J$. Both optimize outer parameters with plain SGD or Adam and assume the model is right, so they supply method rather than competition.
Two papers motivate the problem and constrain the design. Policy Optimization with Differentiable MPC: Convergence Analysis under Uncertainty couples online system identification with plain gradient descent. It identifies dynamics with recursive least squares, a running linear regression. Its convergence relies on that identification becoming exact, which needs persistent excitation. Policy Optimization for Unknown Systems using Differentiable Model Predictive Control is the closest work. It mixes a model-based gradient with a zeroth-order real-system estimate through a hand-scheduled weight, and reaches within 1% of the true optimum on a quadcopter. Our estimator differs structurally. Zheng et al. take a convex combination of two gradients. We use the model only for gradient differences across a shared, measured disturbance, and use real data only as the re-anchoring term. There is no weight schedule to hand-design. Zheng et al.'s use of conservative Jacobians to handle constraint switching also constrains us, as discussed below.
On the variance-reduction side, Stochastic Recursive Momentum for Policy Gradient Methods applies STORM to model-free policies. It handles the replay problem with trajectory importance weights, which its own analysis flags as exploding over long horizons. Our model replay replaces importance weighting entirely. SUSTAIN and FdeHBO show recursive momentum works in bilevel problems. They assume a strongly convex inner problem, which constrained MPC violates when inputs saturate. That is a warning that the smoothness assumption is the weak point.
Research Program
The first component is the estimator and its analysis. We will prove a convergence bound for the update above under two conditions: the true closed-loop cost is smooth, and the model-replay mismatch obeys the $\Delta L$ condition. The bound should show a stationarity rate matching STORM's $\mathcal{O}(\epsilon^{-3})$ rollout complexity, plus a bias floor proportional to $\Delta L\,\eta/\alpha$. We will also show that letting $\eta\to 0$ with $\alpha$ fixed removes the floor.
The second component handles constraint switching. When a parameter change flips which constraints are active in the MPC solve, the model gradient can jump. The replayed difference is then no longer small, which breaks the bias argument. Both replays run in the model, so we can see the active sets at $\theta_t$ and $\theta_{t-1}$ for free. When they differ, the estimator sets $\alpha=1$ for that step, discarding the correction and falling back to the real-data term. We will compare this reset rule against two alternatives: the soft constraint penalties used in DiffMPC, and the conservative Jacobians of Zheng et al.
The third component is an honest accounting protocol. A high-fidelity simulator stands in for "the real system." It is wrapped so that every call draws a fresh disturbance seed and cannot be reset to a previous seed. Only the nominal model may be replayed. Cost is counted in real rollouts, with model solves reported separately as compute.
Baselines
The naive reference point is plain SGD on the zeroth-order real-rollout estimate $g_t$ alone, with the same rollouts per step. It shows how much variance there is to remove.
The strongest existing method is the gray-box hybrid of Zheng et al.. We will reimplement it with its model-based/zeroth-order weighting, using the same nominal model (linear least squares, as in their setup), the same perturbation radius and the same per-step rollout budget. We will tune its weight schedule with the same number of tuning trials we give our $\alpha$ and $\eta$. Zheng et al. report a final cost within 1% of the theoretical optimum on the 12-dimensional quadcopter. No reported result was found on Lacuna for the number of real rollouts it needs to get there. Beating it therefore means reaching that same 1% band in significantly fewer real rollouts, averaged over seeds, without a worse final cost.
The certainty-equivalence variant of Zuliani et al. restricts the MPC to the latest identified model and tunes it with plain gradients. It reports 11% suboptimality on random 4D linear systems, against 30% for their unrestricted variant. On those systems, beating it means lower suboptimality at an equal rollout budget.
Entropy-search Bayesian optimization, following Marco et al., is the low-dimensional black-box competitor. It reported convergence in 20 hardware iterations for 2D and 4D LQR weight tuning, including recovery from a 0.33 m versus 0.64 m pole mismatch. LQR is the linear-quadratic regulator, a classic optimal feedback controller. We will run it at matched rollout budgets on 2D, 4D and higher-dimensional weight spaces.
The decisive ablation sets the replayed correction to zero. This reduces our estimator to an exponential moving average of $g_t$, which is ordinary momentum. A second ablation computes the correction with fresh model noise instead of the reconstructed $\hat\xi_t$. That isolates the value of sharing the measured disturbance. An oracle that replays the real simulator with a common seed gives an upper bound that is unattainable on hardware.
Why not just…?
Why not just use larger batches of real rollouts or ordinary momentum? Averaging $B$ rollouts cuts variance by $1/B$ at $B$ times the cost. Ordinary momentum averages stale gradients and lags behind a moving iterate. That lag is exactly what the recursive correction removes. The zero-correction ablation is ordinary momentum at matched budget. If it matches our method, the proposal fails, and the experiments will report that plainly.
Why not just trust the model gradient, perhaps with online identification? Zheng et al. show model-only gradients converge to the model's optimum, not the system's. Zuliani et al. get good results only when identification is asymptotically exact. Even then the certainty-equivalence variant sits at 11% suboptimality on random linear systems. We will sweep model mismatch from realizable to clearly unrealizable, for example unmodeled drag on the quadcopter. The prediction is that model-only methods plateau at a mismatch-dependent cost while ours does not. If the identified model is already near exact, the model gradient will rightly be competitive. We would treat that regime as one where our method should not be needed.
Why not just use Bayesian optimization, as Marco et al. did? In four or fewer dimensions, 20 trials is a hard number to beat, and we expect it to win or tie there. Our claim concerns the regime beyond about ten parameters, such as full diagonal state and input weights plus horizon-wise terms on the quadcopter. There, Gaussian-process surrogates scale poorly and discard per-timestep information. The dimension sweep will locate the crossover. If none appears below 30 dimensions, the claim narrows.
Why not just apply STORM-PG with importance weights? It needs a stochastic policy with tractable trajectory likelihoods. Its weights become unstable over long horizons, which its own analysis acknowledges. An MPC is deterministic, so we would have to add exploration noise. We will include this variant on the cart-pole task, where STORM-PG reported reaching maximum reward in about 500 trajectories versus 1,500 for SRVRPG.
Proposed Experiments
The main testbed is the 12-dimensional nonlinear quadcopter with rotor and tilt constraints, used by both Zuliani et al. and Zheng et al. Its "real system" is the full nonlinear simulator with stochastic wind, and its nominal model is identified by least squares. The secondary testbeds are Zuliani et al.'s random 4D linear systems, with their kinematic bicycle racing line (35% tracking cost reduction reported there), and a simulated pole-balancing LQR task reproducing Marco et al.'s pole-length mismatch. Gradients through the MPC use the DiffMPC solver. The primary metric is real rollouts to reach within 1% and 5% of the true optimal closed-loop cost, which is computable in simulation by tuning against the full model. Secondary metrics are final suboptimality, the variance of $G_t$ against a large-batch reference gradient, and the frequency of constraint violations during tuning. Each configuration runs over 20 seeds.
Four checks decide the outcome. First, the full method must beat the zero-correction ablation and the Zheng et al. hybrid in rollouts-to-1% on the quadcopter. Second, as model mismatch grows, its final cost must stay near optimal while the model-only and certainty-equivalence baselines plateau. Third, the measured tracking bias must shrink roughly linearly as $\eta$ decreases at fixed $\alpha$, as the bias bound predicts. Fourth, removing the active-set reset must produce visible variance spikes at constraint switches, confirming that the smoothness condition is the operative risk. If the second or third check fails, the core mechanism is wrong. In that case the model-replayed difference is not more trustworthy than the model gradient itself.
References
- Momentum-Based Variance Reduction in Non-Convex SGD
- SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient
- Differentiable MPC for End-to-end Planning and Control
- Differentiable Model Predictive Control on the GPU
- Policy Optimization with Differentiable MPC: Convergence Analysis under Uncertainty
- Policy Optimization for Unknown Systems using Differentiable Model Predictive Control
- Stochastic Recursive Momentum for Policy Gradient Methods
- A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentum
- Achieving O(ε-1.5) Complexity in Hessian/Jacobian-free Stochastic Bilevel Optimization
- Automatic LQR tuning based on Gaussian process global optimization