Training the Planner the Robot Actually Runs: Deadline-Matched Differentiable MPC

Model predictive control (MPC) is a controller that, at every control tick, solves a short optimization problem to plan the next few seconds of motion, executes the first action, and then replans. A differentiable version lets a learning algorithm tune the planner's cost function and internal dynamics model by gradient descent. On a real robot, though, the planner almost never finishes solving. A 100 Hz loop leaves a few milliseconds, so the solver runs two or three iterations, starting from the previous tick's plan, and whatever it has is sent to the motors, several milliseconds late. The standard way to differentiate MPC, Differentiable MPC for End-to-end Planning and Control, computes gradients by assuming the solver reached its exact optimum. That assumption fails in precisely this regime. The result is that the planner being trained is not the planner being deployed.

We propose to train the deployed object directly. The truncated, warm-started, delayed planner is treated as a recurrent stochastic policy: its "memory" is the plan carried from one tick to the next. Its cost and model parameters are trained with a policy gradient estimator that uses the planner's own noiseless plan as a baseline. The bet is that matching the training computation to the deployment deadline matters most when the solver budget is smallest and the simulator is most wrong. That is the combination that governs sim-to-real transfer. The planner's built-in lookahead should also keep gradient variance low enough that a handful of interpretable parameters can be learned from far fewer rollouts than a reactive network needs.

Core Claim

Fix an iteration budget $K$ dictated by the control deadline. At tick $t$, the controller runs $K$ iterations of iterative LQR (a Newton-like trajectory optimizer), starting from the previous plan shifted forward one step:

$$(\mu_t, z_t) = \Phi^K_\theta\big(s_{t}, \mathrm{shift}(z_{t-1})\big), \qquad a_{t+\delta_K} \sim \mathcal{N}(\mu_t, \sigma^2 I).$$

Here $s_t$ is the observed state and $z_t$ is the current planned trajectory, which acts as the recurrent state. $\theta$ collects the learnable cost weights and dynamics-model parameters, $\mu_t$ is the commanded action, and $\delta_K$ is the actuation delay implied by spending $K$ iterations on a target CPU. The claim is that tuning $\theta$ through this exact map has two consequences:

  • It yields a controller that beats the same architecture trained at convergence and then truncated at deployment.
  • That margin widens as $K$ shrinks and as the gap between simulator and test dynamics grows.

The reason is that the truncated planner trained this way learns cost shapes whose few-iteration solutions are good. A converged-trained planner only learns cost shapes whose optima are good.

This differs from the closest work in specific ways. Policy Optimization with Differentiable MPC: Convergence Analysis under Uncertainty proves convergence of gradient-based MPC tuning with online system identification. It reports 11% suboptimality for its certainty-equivalence variant versus 30% without it on random 4D linear systems. However, it assumes converged solves and explicitly leaves "anytime feasibility" under real-time limits open. Autotuning Bipedal Locomotion MPC with GRFM-Net tunes a real-time MPC on hardware. It cuts weighted loss by up to 68.7% versus nominal and 40.5% versus expert tuning, but it optimizes a supervised tracking loss offline and does not model solver truncation or delay. Policy Gradient with Tree Expansion motivates the variance side. It proves that embedding lookahead in the policy shrinks gradient variance exponentially in depth, and it bounds the bias that grows with depth under a wrong model. Its experiments are discrete Atari with an exact simulator and no latency cost. We carry that variance-versus-bias tension into continuous control, where depth also costs wall-clock time.

Research Program

Choosing the gradient. Under a deadline, $K$ is small, typically 1 to 5. Backpropagating through the solver iterations within a tick is therefore cheap. The memory objection that Amos et al. raise against unrolling, linear growth in iterations, mostly disappears exactly when deadlines bind. The harder graph is the chain of warm starts across ticks, which spans the whole episode. Long unrolled chains of this kind become chaotic, as ES-Single documents. We will therefore compare three treatments of the cross-tick link:

  • full backpropagation through the warm-start chain;
  • a stop-gradient on the shifted plan;
  • a score-function (likelihood-ratio) estimator over the exploration noise, using the planner's pathwise gradient only within a tick.

Reducing variance with a self-critical baseline. For the score-function part we use a baseline in the spirit of Self-Critical Sequence Training, which subtracts the reward of the model's own greedy output instead of training a critic:

$$\hat g = \nabla_\theta \log \pi_\theta(\tau)\,\big(R(\tau) - R(\bar\tau)\big).$$

Here $\tau$ is a rollout with exploration noise and $\bar\tau$ is the noiseless rollout of the same truncated planner. Both are run in simulation with the same random seed for randomized physics and sensor noise. Sharing the seed makes $\bar\tau$ a strongly correlated control variate. We will measure variance directly, as the trace of the covariance of $\hat g$ across independent batches.

Mapping the trade-off. We will sweep planning horizon $H$, iteration budget $K$, and the size of the dynamics shift between training and test. At each point we record three quantities:

  • gradient variance;
  • gradient bias, measured against a high-sample reference gradient computed in the test dynamics;
  • per-tick latency.

The prediction from Dalal et al. is that the best horizon shortens as model error grows. Our addition is that latency imposes a second ceiling on $H\cdot K$ that is independent of model error. The deliverable is an empirical map of where each constraint binds.

Baselines

Reactive policy with domain randomization. The naive reference is a reactive neural policy trained with PPO under domain randomization over masses, frictions, motor gains, and control delay, as in Control of a Quadrotor with Reinforcement Learning. It has negligible inference latency, which is the property our method gives up. Fairness rules:

  • It receives the same simulator, the same randomization ranges, and the same environment-step budget.
  • It gets a hyperparameter search of equal size.
  • Its training is not capped by environment steps alone, so that sample efficiency is not the only axis compared.

Beating it means higher return and fewer constraint violations under held-out shifts at equal environment steps. I found no reported Lacuna result for PPO with domain randomization on these exact tasks. The only cited figure is the up-to-three-orders-of-magnitude variance gap between flat PPO and SoftTreeMax on Atari.

Converged-then-truncated differentiable MPC. The strongest existing method is the same MPC architecture trained with implicit gradients at convergence, following Amos et al.. Where online identification applies, we add the certainty-equivalence system identification of Zuliani et al.. At test time it is deployed with the same $K$, warm start, and delay as ours. For this baseline the strongest reported figures on Lacuna are 11% suboptimality on random 4D linear systems and roughly 35% tracking-cost reduction on kinematic-bicycle racing, both from Zuliani et al. We reproduce those settings first and confirm we recover these numbers before comparing. This comparison also serves as the key ablation, because the only difference from our method is whether training matched the deployment computation. Beating it means lower closed-loop cost at every $K$ below convergence, with the margin growing as $K$ decreases.

Evolution strategies on the planner parameters. The third baseline applies ES-Single to $\theta$. Its weakness on large networks is cost that grows with parameter count, but that weakness does not apply here because $\theta$ has tens to hundreds of entries. It uses the same rollout budget. It is unbiased and has constant variance across long unrolls, so it is the honest test of whether our gradient machinery is needed at all.

Ablations of the variance tools. Two further ablations isolate the variance components: removing the self-critical baseline, and replacing it with a learned value critic.

Why not just…?

Why not just train a reactive network with heavy domain randomization? It is fast at deployment and well proven. The file reports SAC settling for sitting up on Humanoid Standup where planner-guided policies stood (APEX). It also reports SURREAL manipulation tasks unsolved with 64 actors (SURREAL). Both point to exploration failures that lookahead mitigates. Neither shows a planner winning under dynamics shift, however, so this is a genuine threat. The held-out-shift experiment settles it, and on smooth low-contact tasks we expect domain randomization to tie.

Why not distill a converged planner into a fast network? APEX does exactly that with iCEM and DAgger and removes latency entirely. The distilled network cannot replan when the real dynamics differ from training. A truncated planner still re-solves from the true current state each tick. We include APEX-style distillation of our converged MPC as an extra comparison. If it matches us under shift, the case for online planning collapses.

Why not just use implicit gradients and accept the mismatch? If truncated solves are already near-converged because warm starts are good, implicit gradients may be accurate enough. This is the main falsification risk. The $K$ sweep against the converged-then-truncated baseline measures it directly. If the gap vanishes at realistic $K$, we report that as a negative result.

Why not compensate delay with a history-conditioned belief network instead? DT-CORL handles 16- to 64-step delays with a Transformer belief state. For example, it scores 71.8 versus 42.9 for belief-based CQL on Hopper-medium at a 16-step delay. It addresses observation and actuation delay, not the planner's own compute delay. It is also complementary rather than competing, since it could supply the planner's state estimate. We test the combination on the D4RL delayed Hopper setting.

Proposed Experiments

Tasks. We use five settings:

  • pendulum and cartpole swing-up from Amos et al.;
  • the 12-dimensional nonlinear quadcopter and kinematic-bicycle racing from Zuliani et al.;
  • MuJoCo HalfCheetah and Hopper, to introduce contact;
  • robosuite Lift (robosuite) as a manipulation stretch goal.

Sim-to-real proxy. Because no hardware is assumed, transfer is tested sim-to-sim. Training uses nominal parameters with randomization. Testing uses held-out friction, mass, and motor-lag shifts outside the training range. At test time, actuation delay equals the measured wall-clock time of $K$ iterations on a fixed embedded-class CPU. Where possible we use a different integrator or contact model at test time. We will state plainly that this proxy cannot capture every hardware effect. If a quadrotor platform becomes available, it is the first hardware target.

Metrics. We report:

  • closed-loop return or tracking cost;
  • constraint-violation rate on torque and obstacle limits;
  • gradient variance and bias per update;
  • per-tick latency;
  • environment steps to reach a target return.

Decisive checks.

  • At $K\in\{1,2,3,5\}$, deadline-matched training must beat converged-then-truncated training. The margin must shrink toward zero as $K$ approaches convergence, which tests that the mechanism is truncation and not extra tuning.
  • The self-critical baseline must measurably reduce variance relative to no baseline, and must not be worse than a learned critic.
  • On held-out shifts, our method must beat reactive domain randomization and APEX-style distillation on violation rate at matched latency.
  • The best $H$ in the trade-off map must shorten as the dynamics shift grows, as predicted by Dalal et al.'s bias bound. If it does not, that bound's relevance to continuous control is in doubt.