A Reality Check for Every Step: Sufficient-Decrease Tests That Let Model Predictive Control Back Off When the Simulator Lies

A controller that plans with a simulator always makes promises. At each moment it predicts that if it applies a certain action, the task will get a certain amount closer to done. Model predictive control (MPC) works this way. It is also called receding horizon control: it re-plans a short sequence of actions at every time step using an internal model, applies only the first action, observes the new state, and plans again. When that internal model is a simulator that differs from the real machine, the gap between simulation and reality (the sim-to-real gap), the promises stop matching what happens. Re-planning from the observed state corrects some of this. It does nothing, though, about the planner trusting the model just as much on the next step. We propose to borrow the central test of backtracking line search, the Armijo sufficient-decrease condition. In optimization it accepts a step only if the objective actually falls by at least a fixed fraction of what a local model predicted. We apply it across time on the real system: after each executed action, check whether reality delivered at least a fraction $c_1$ of the progress the simulator promised. If it did not, shrink how much the controller defers to the simulator-optimal plan. If it did, for several steps in a row, grow that trust back.

The opportunity is practical. A deployed MPC controller already computes everything this test needs: the optimal planned cost at each state, and the model's one-step prediction. The test therefore adds almost no computation, and it needs no gradients, so it works with sampling-based planners in contact-rich settings where Armijo's usual derivative is undefined. The smoothed test statistic is also an online, task-relevant measurement of how large the sim-to-real gap is right now, which is more useful to an operator than a parameter estimate.

Core Claim

Let $J(x)$ be the optimal cost the MPC planner computes from state $x$ over its horizon. Let $\ell(x,u)$ be the cost of one step taken with action $u$, and let $\hat{x}_{t+1}$ and $x_{t+1}$ be the model-predicted and the actually observed next states. Standard MPC stability arguments treat $J$ as a Lyapunov function, a quantity that must fall at every step when the model is correct. The promised decrease is $d_t = J(x_t) - \ell(x_t,u_t) - J(\hat{x}_{t+1})$ and the realized decrease is $r_t = J(x_t) - \ell(x_t,u_t) - J(x_{t+1})$. Both are available as soon as the next plan is solved. Our acceptance test is

$$\bar{r}_t \ge c_1\, \bar{d}_t ,$$

where the bars denote exponential moving averages over recent steps and $c_1 \in (0,1)$. The executed action is $u_t = u^{\text{rob}}_t + \alpha_t\,(u^{\text{mpc}}_t - u^{\text{rob}}_t)$. Here $u^{\text{mpc}}_t$ is the simulator-optimal action, $u^{\text{rob}}_t$ comes from a conservative prior policy trained with dynamics randomization, and $\alpha_t \in [0,1]$ is the trust step. When the test fails, $\alpha$ is contracted by a factor, exactly as a backtracking line search contracts its step. When it passes for $k$ consecutive steps, $\alpha$ is expanded.

The claim is that this cross-time backtracking keeps nearly all of MPC's performance when the model is right, falls back gracefully when it is wrong, and does better than any fixed blend or fixed horizon tuned on the gap distribution. It should also do better than pure MPC and pure robust policies under gaps that cannot be identified, such as latency and unmodeled elasticity.

The mechanism is structurally a trust-region ratio test, which compares actual to predicted reduction, and we say so plainly. The new parts are three. The "function evaluation" is a single noisy step of a physical system, so it cannot be repeated at several step sizes the way an optimizer can. The predicted decrease comes from a planner's value function rather than a gradient. And the variable being backtracked is trust in the simulator rather than distance in parameter space.

Grounding in Prior Work

Several papers supply the method. The Armijo condition and its convergence theory in Armijo Line-search Makes (Stochastic) Gradient Descent Go Fast show that adapting step size to local geometry can change convergence qualitatively. That result motivates letting $\alpha$ track the local reliability of the model instead of fixing it for the worst case. No learning rates needed: Introducing SALSA - Stable Armijo Line Search Adaptation supplies the specific stabilizer we reuse. It smooths both sides of the Armijo inequality with moving averages so that mini-batch noise does not collapse the step size, and it checks the condition infrequently when steps are stable. In our setting, sensor noise and stochastic disturbances play the role of mini-batch noise. Our test applies SaLSa's smoothing to the closed loop instead of to training. Arjevani et al. show that step schedules fixed in advance face lower bounds that adaptive ones escape. This is the theoretical reason to expect a fixed tuned blend to underperform.

Other work motivates the problem. Bootstrapped Model Predictive Control had to restrict temporal-difference learning to a single model step because errors in its learned model compounded. That is direct evidence that how far to trust a model should be adjusted rather than assumed. Goldschmidt et al. show that receding horizon updates keep a controller from staying anchored to wrong initial modeling assumptions under drift. We take the next step: update not only the plan but the controller's trust in the planner. Micheli et al. show that receding horizon feedback already absorbs large forecast errors. This constrains our claim, because under mild gaps plain MPC may be enough, and our method must not do worse there. Lu et al. show that noisy look-ahead still reduces the gap to optimal. This argues against giving up on the model entirely, and it is why $\alpha$ only shrinks toward the robust policy rather than switching to it.

The sim-to-real literature supplies the gap sources and the fallback. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization and Sim-to-Real: Learning Agile Locomotion For Quadruped Robots train policies that survive randomized physics and latency. We use such a policy as $u^{\text{rob}}$ rather than competing with it. MuJoCo MPC for Humanoid Control: Evaluation on HumanoidBench supplies the planner: iterative linear-quadratic and sampling-based MPC. It also reports a failure mode that the test should catch. Short-horizon MPC exploits the end of its planning window, for example by throwing the robot's center of mass forward and collapsing. The model's promised decrease then far exceeds what the next real state supports.

A caveat on coverage: the research file available to us was truncated partway through the sim-to-real methods. It contained no section surveying existing combinations of these directions. We found no paper in the file that applies a sufficient-decrease test across time to gate trust in a simulator during MPC, but we cannot rule out such work beyond the file.

Research Program

We will build the controller on top of MuJoCo MPC's planners. The first component is the test itself. It needs $J$ at the observed and the predicted next state. The observed value is the next planning solve. The predicted value costs one extra planner evaluation, warm-started from the same solution. We will measure the overhead and compare it with the planning budget.

The second component is the adaptation rule. It has a contraction factor, an expansion window $k$, $c_1$, and the smoothing rate. We will analyze its behavior under a simple model in which realized decrease equals promised decrease times an unknown, slowly varying reliability factor plus noise. The goal is to state conditions under which $\alpha$ tracks the largest blend that still guarantees per-step Lyapunov decrease. This mirrors how the Armijo analysis in Armijo Line-search Makes (Stochastic) Gradient Descent Go Fast ties accepted steps to local smoothness.

The third component is a second "knob." Instead of blending actions, a failed test shortens the planning horizon $N$. This follows the evidence in Bootstrapped Model Predictive Control that shorter model rollouts are safer when the model is wrong. We will report which knob, or which combination, works better.

We will also study the smoothed ratio $\bar{r}_t/\bar{d}_t$ as a gap detector on its own: how early it falls before a fall or task failure, and how it correlates with injected gap magnitude.

Baselines

The naive reference point is plain MPC with $\alpha = 1$, using the nominal simulator and the same planner and compute budget. It shows how much receding horizon feedback alone absorbs the gap, which Micheli et al. suggest can be a lot. Beating it means lower cumulative real cost and fewer failures under gap. Equally important, it means no measurable loss when there is no gap.

The second baseline is the robust prior alone ($\alpha = 0$), a policy trained with dynamics randomization as in Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. It is the standard sim-to-real answer and the strongest competitor under large gaps.

The third and most important baseline is a fixed blend and fixed horizon. Both are tuned by grid search on the same gap distribution, with a tuning budget equal to what our method uses to set its hyperparameters. This isolates the adaptive backtracking as the key ingredient. If a well-tuned constant $\alpha$ matches us, the idea fails.

The fourth baseline is MPC with online system identification. It uses an extended Kalman filter that re-estimates model parameters, in the style of Adaptive Digital Twin Modeling with Control: Integration of Extended Kalman Filter-Based Recursive Sparse Nonlinear Identification with Model Predictive Control. It receives the same parameter families we perturb.

For the strongest existing method, MuJoCo MPC on HumanoidBench, the file reports only a speed figure: about 185 times slower than real time on consumer hardware (MuJoCo MPC for Humanoid Control: Evaluation on HumanoidBench). We found no reported task-success or return figure on Lacuna for MuJoCo MPC under sim-to-real or sim-to-sim gaps. We will therefore establish all baseline numbers ourselves under identical seeds, compute, and gap draws.

The ablations replace the smoothed test with an unsmoothed one, to test whether SaLSa-style averaging matters. They replace backtracking with a heuristic that reacts to raw state-prediction error, to test whether measuring error in task-cost units matters. And they compare the blend knob with the horizon knob.

Why not just…?

Why not just use dynamics randomization and skip the model at deployment? A randomized policy is robust because it is conservative across the whole range of physics it was trained on. That costs performance on any particular system that sits inside the range, which is the oblivious-schedule limitation Arjevani et al. describe, applied to control. Our comparison against $\alpha = 0$ on small and zero gaps will show whether keeping the planner recovers that lost performance. If it does not, the method has no reason to exist.

Why not just identify the real parameters online and keep full trust in the corrected model? For gaps that are parametric and identifiable, such as mass and friction, this is a real threat and may win outright. It cannot fix gaps outside the model's structure: latency, actuator dynamics, elasticity, or contact behavior the rigid-body simulator does not represent. Our experiments split gaps into these two families, and we expect the advantage to appear only in the second. We will also test identification combined with our gate, since the two do not conflict.

Why not just tune a fixed conservative blend or a short horizon? It is the simplest competitor. If gaps were stationary, it might suffice. The test is the drifting-gap condition, where the payload or friction changes mid-episode. A single tuned constant must then be wrong for part of the episode. The tuned-constant baseline in this condition is our decisive comparison.

Proposed Experiments

All main experiments are sim-to-sim. The planner uses nominal MuJoCo models. The "real" plant is a perturbed copy with altered masses and friction, injected action latency of one to three control steps, reduced actuator gains, added joint compliance, and observation noise, drawn from distributions disjoint from the randomization ranges used to train $u^{\text{rob}}$. Because planning time does not affect sim-to-sim results, the reported slow speed of MuJoCo MPC does not block humanoid tasks. We will run a quadruped locomotion task modeled on the setting in Sim-to-Real: Learning Agile Locomotion For Quadruped Robots and a subset of HumanoidBench locomotion and reaching tasks. We will include the reach task where window-end exploitation was observed.

Metrics are cumulative task cost on the perturbed plant, failure or fall rate, the fraction of runs in which the per-step Lyapunov decrease holds, and the planning overhead added by the test.

Four results settle the claim. First, with zero gap, $\alpha$ should stay near one and performance should match plain MPC within seed noise. Second, as gap magnitude grows, our method's cost curve should lie below the lower envelope of plain MPC, the robust policy, and every fixed tuned blend. Third, under mid-episode gap changes, $\alpha$ should track the change, and its advantage over the best fixed blend should be largest there. Fourth, a fall in the smoothed ratio should precede failures by a measurable lead time and should flag the window-end exploitation behavior on HumanoidBench reaching. If resources permit, a small hardware check on a quadruped would test whether the sim-to-sim conclusions hold on a physical robot. Hardware is a confirmation, not the basis, of the claims.