Training Anytime Diffusion Policies Under Randomized Deadlines

Fast robot policies built on diffusion models, which generate an action sequence by repeatedly removing noise from a random guess, have recently been made fast enough for real robots. The main trick is warm-starting: each new plan begins from the unexecuted remainder of the previous plan instead of from pure noise, and is then cleaned up in only one to three steps. All of these accelerators assume that the robot always gets its fixed number of cleanup steps, and that the world changes smoothly between plans. Both assumptions fail on real hardware. Compute stalls, operating-system jitter, and shared GPUs make the available time vary from cycle to cycle. Slips, bumps, and occlusions break the smoothness that warm-starting relies on. The policy is then either late or acting on a stale plan.

We propose to train the policy in simulation under the conditions that break it. Each control cycle in a training rollout draws a random compute budget (how many cleanup steps finish before the deadline) and a random actuation delay, alongside the usual randomized friction, masses, and visuals. The robot executes whatever plan exists when the deadline hits. A privileged expert then relabels the states those truncated plans led to. The denoiser is trained so that stopping after any number of steps, including zero, yields an executable plan, and so that each extra step makes it better. The result should be a policy whose quality degrades gracefully when time is short, which uses spare time when it has it, and which transfers zero-shot to a real arm with real timing noise.

Core Claim

The core claim is that anytime behavior in a neural controller has to be trained on the state distribution produced by its own truncated computation, not added at inference time. Inference-time accelerators such as Real-Time Iteration Scheme for Diffusion Policy and Falcon never see the states that arise when a two-step plan is executed in place of a ten-step plan under a perturbation. Those are exactly the states where warm-start contraction fails. We claim that closing this loop in simulation, with compute budget and latency randomized like any other physical parameter, produces better success-versus-budget curves and better zero-shot transfer than applying those accelerators to a policy trained on the same randomized data.

Mechanism

The action chunk is represented as a clamped B-spline, following BEAST, which fits demonstration chunks onto splines whose first control points are pinned to the end of the previous chunk. We reuse this representation only for its structural guarantee. After every denoising step we re-clamp the first control points to the currently executing segment and clip the rest to joint limits. Every intermediate plan is therefore continuous with what the robot is already doing and kinematically legal. This is a weak validity guarantee: it says nothing about task success. That gap is what training must close.

The step-zero plan is the shifted tail of the previous chunk, the same warm start RTI-DP uses. It plays the role of the constant-time incumbent in Constant-time Motion Planning with Anytime Refinement for Manipulation. That work retrieves a guaranteed-valid geometric path instantly and then polishes it with an ARA*-style search, a search whose solution quality is bounded by an inflation factor shrunk toward 1 as time allows. Our incumbent is free, needs no 126-minute precomputation, and works from pixels. The price is that it carries no optimality bound.

Training starts from a standard Diffusion Policy fit to demonstrations in a dynamics- and visually-randomized simulator. It then proceeds through deadline-randomized rollouts. At each cycle we sample a budget $k$ from $\{0,\dots,K\}$ out of a heavy-tailed distribution and a delay $d$, and execute the plan after $k$ steps. At visited states, a privileged expert supplies a target chunk $a^*$. This follows the asymmetric idea of Asymmetric Actor Critic for Image-Based Robot Learning, which gives the training-side model ground-truth state while the deployed actor sees only images. Here the expert is either the full-step teacher given true state, or a scripted or motion-planning expert where one exists. The denoiser is trained with

$$ \mathcal{L} = \sum_{k=1}^{K} w_k \lVert \hat{x}_k - a^* \rVert^2 + \lambda \sum_{k=0}^{K-1} \max\!\left(0,\ \lVert \hat{x}_{k+1} - a^* \rVert^2 - \lVert \hat{x}_k - a^* \rVert^2\right). $$

Here $\hat{x}_k$ is the projected plan after $k$ steps, with $\hat{x}_0$ the shifted incumbent. The weights $w_k$ put mass on small $k$ so that early stops are useful. The hinge, scaled by $\lambda$, penalizes any step that moves the plan away from the expert. The hinge directly targets the non-monotonicity of stochastic samplers that classical anytime search avoids by construction.

New rollouts are generated with the updated policy, as in DAgger-style dataset aggregation. Over iterations, the training data comes to include the drift caused by under-budgeted, delayed actions under randomized physics. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization randomized the control timestep as a passive environment parameter for a single-step recurrent policy. We make the budget an active variable that shapes the policy's own computation.

Research Program

The work has three parts. First, we build the training loop on ManiSkill2 and Robomimic with randomized friction, mass, damping, sensor noise, textures, and lighting, plus a compute model. The compute model samples budgets and delays from distributions measured on our target edge GPU under background load, then widened by a factor of two. Second, we characterize anytime profiles: success as a function of a fixed budget $k$, and success under randomized deadline distributions, including bursts of zero-step cycles. Third, we deploy zero-shot on a physical arm, with real timing jitter and injected perturbations such as pushes and object swaps.

A secondary analysis asks how deadline-randomized training reshapes the region of warm starts from which a few steps recover a good plan. We measure this region empirically by perturbing incumbents by controlled amounts and recording how much error remains after $k$ steps. We then compare it to the local contractivity condition stated by RTI-DP. That paper proves errors shrink exponentially only when the warm start lies inside such a region, and our hypothesis is that training enlarges it.

Baselines

The naive reference point is a full-step Diffusion Policy that blocks until its plan is ready, so the robot holds the stale chunk on overrun. We also run the same policy with a fixed small number of denoising steps from noise. On Push-T, RTI-DP reports that standard Diffusion Policy takes 816 ms per inference, which sets the scale of the problem.

The strongest existing method is RTI-DP applied to a Diffusion Policy trained on exactly our randomized simulation data. Its inference is truncated to the sampled budget, and it takes the shifted incumbent when the budget is zero. It is the strongest competitor because it shares our warm start, needs no retraining, and is reported to cut Push-T latency from 816 ms to 25 ms with "comparable overall performance" to full-step denoising. The file contains no itemized RTI-DP success rates, and no reported result was found on Lacuna for any method under randomized deadlines.

Consistency Policy distills the diffusion teacher into a model that needs only one to three steps. It reports 77% success on Tool Hang at 3 steps against the 100-step teacher's 79%, and about 21 ms on a laptop GPU. We distill it from the same randomized-data teacher. Falcon reuses cached partially-denoised samples from earlier cycles and falls back to full denoising when none are close enough; it reports hardware latency dropping from 0.43 s to about 0.15 s. We run it with its threshold tuned on our validation tasks. STEP trains a small predictor to propose the warm start and then refines it in 2 to 4 steps. It reports 20 ms on an RTX 2050 and 21.6% higher success than BRIDGER. We run it with its released code (Kimho666/STEP). Finally, a Peng-style recurrent policy trained with dynamics and timestep randomization serves as the single-step reference.

All methods use the same demonstrations, the same randomized simulator, and the same backbone where applicable. Every method's total training compute, including our rollout iterations, is matched by giving baselines extra epochs or extra distillation data. Each method receives the same hyperparameter-search budget. All are evaluated on the same deadline distributions on the same hardware.

Two ablations isolate the key ingredient. The first is our full loss with rollouts at a fixed budget, which isolates budget randomization. The second removes the hinge, which isolates monotonicity. Further ablations drop the spline projection and the dynamics randomization. Beating the field means two things. Our area under the success-versus-budget curve must exceed RTI-DP-on-randomized-data with non-overlapping confidence intervals across seeds. Our success at $k=1$ under perturbation must also exceed Consistency Policy at its native budget, with no p99 latency spike.

Why not just…?

Why not just distill to a fixed few-step Consistency Policy, which always meets the deadline? This is a real threat. A 3-step model losing only two points on Tool Hang suggests that, in nominal conditions, refinement buys little. Its weaknesses are that it cannot output anything if interrupted mid-chain, it cannot use spare time, and distillation narrows the range of distinct behaviors. None of these matter if deadlines are always met and tasks have a single mode. Our experiments therefore include a nominal-deadline condition and the multimodal block-pushing task used by RTI-DP. If Consistency Policy matches us there and under randomized deadlines, we will report that anytime training is unnecessary for those regimes.

Why not just train on randomized simulation data and apply RTI-DP or Falcon at inference? This is the central comparison, and it is built into the Baselines. Falcon's documented fallback spike to 0.43 s shows that inference-only methods mishandle the off-smooth cases. However, RTI-DP with a zero-step fallback avoids spikes entirely. If the fixed-budget-rollout ablation and RTI-DP-on-randomized-data match our full method, the claim that training on self-induced drift matters is falsified, and we would narrow the contribution to the projection and hinge.

Why not just randomize latency for a fast single-step recurrent policy, as Peng et al. did? That approach reaches 0.89 real success on puck pushing with constant latency. However, Diffusion Policy reports a 46.9% average improvement over LSTM-GMM-style policies on multi-step manipulation, which suggests single-step recurrent policies lack the capacity for our tasks. We include the recurrent policy as a baseline to measure whether that gap persists under timing noise.

Proposed Experiments

In simulation, we use Robomimic Square, Transport, and Tool Hang, Push-T, and ManiSkill2 StackCube, TurnFaucet, and PushChair. We report anytime profiles over $k$ from 0 to 8, success under three deadline distributions (nominal, jittered, and bursty), deadline-miss rate, p99 latency, jerk at chunk boundaries, and mode coverage.

For zero-shot transfer, we use a physical arm on a pick-cube task and a real Push-T setup. The reference points are ManiSkill2's reported 91% simulated and 60% real zero-shot success on PickCube, and Diffusion Policy's 95% on real Push-T, which was trained on real demonstrations rather than zero-shot. We inject scripted pushes during execution and load the GPU with background work to induce real jitter.

The decisive check is a two-way comparison on the real robot under induced jitter. We compare our method against RTI-DP on the same randomized-data policy and against the fixed-budget ablation, varying perturbation on or off. The proposal stands only if our advantage grows in the perturbed, jittered cell, which is where self-induced drift should matter most.