Robots trained by imitation copy their demonstrations, including the hesitations and detours. A growing line of work improves them afterward. A frozen imitation policy proposes several candidate motions, and a learned scorer, trained only from whether past attempts succeeded, picks among them. Ranking alone has a hard limit: it can only choose motions the imitation policy already proposes. It also has a hidden hazard. When the scorer averages candidates that go around an obstacle on opposite sides, the average can drive straight through the obstacle.
We propose to spend the same per-step compute budget differently. The policy draws a few candidates. Each one gets a small number of gradient steps on the learned scorer, kept close to its starting point and projected onto the robot's joint, velocity, and obstacle constraints. Then exactly one refined candidate is executed, never a blend. The central question is empirical and has a clean answer under a matched wall-clock deadline. Does a little local optimization per candidate beat simply drawing more candidates? Does selecting one refined candidate, instead of averaging, remove constraint violations without giving up the self-improvement that value-guided ranking already delivers?
Core Claim
The direction this builds on most directly is Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning. It couples a frozen vision-language-action policy with a compact Q-function, a critic that predicts the chance of eventual success from the current state and a proposed action chunk. The critic is trained on autonomous rollouts with only a binary success reward, and the robot executes a Q-weighted average of sampled action chunks. This raises LIBERO-10 success from 93% to 99% and RoboTwin from 83.8% to 91.4%. The authors note two limits. The critic cannot promote actions the frozen policy never proposes, and the method enforces no physical constraints.
Our claim is that both limits can be addressed at test time, inside a real-time budget, without retraining the base policy and without a dynamics model. The method rests on three choices. First, each candidate is refined separately by trust-region ascent on the critic, which extends behavior a short distance beyond the prior's support. Second, a single refined candidate is selected, so the motion stays inside one way of going around obstacles (one homotopy class). Third, the selected chunk is projected onto the constraint set. That projection becomes a small convex quadratic program (QP) because the selection has already fixed which side of each obstacle to use, so obstacle constraints can be linearized as half-spaces.
In a fixed deadline $T$, the controller chooses the number of proposals $N$ and refinement steps $K$ subject to
$$ N \cdot (c_{\text{sample}} + K\, c_{\text{grad}}) + c_{\text{QP}} \le T, $$
where $c_{\text{sample}}$ is the cost of one proposal, $c_{\text{grad}}$ the cost of one critic-gradient step, and $c_{\text{QP}}$ the cost of the final projection. Ranking-only methods sit at $K=0$. We predict that the best $K$ is greater than zero, and that it grows as the prior becomes more confident (lower entropy), since extra samples then mostly repeat each other.
Research Program
The base policy is a pretrained Diffusion Policy, a generator that produces action chunks by gradually denoising random noise (Diffusion Policy). Its sampling is accelerated with the warm-started denoiser of STEP, which reaches 20 ms end-to-end latency on an RTX 2050. This makes $c_{\text{sample}}$ small enough for the trade-off to be real. STEP supplies speed only; on its own it cannot exceed the demonstrations.
The critic follows the Q-planning recipe: categorical (HL-Gauss) regression on sparse binary completion rewards over the robot's own rollouts. We use the expected value of the categorical output, which gives smoother gradients than the logits.
Refinement takes $K$ steps of gradient ascent on the critic with respect to the action chunk. Each step is followed by projection onto an $\ell_2$ ball of radius $\epsilon$ around the original sample. Optionally we add a penalty from the diffusion model's own score, so refined actions stay where the prior has some density. This trust region is the safeguard against the critic rewarding actions it has never seen.
After refinement, we select the candidate with the highest critic value. We then solve one QP for the nearest chunk that satisfies box limits on joints and velocities and linearized obstacle half-spaces. The QP is warm-started from the refined chunk itself. Prior work on learned QP warm-starts shows such initializations can cut solver iterations sharply, for example from over 600 to 48 on a vehicle-dynamics problem (End-to-End Learning to Warm-Start for Real-Time Quadratic Optimization).
Executed refined actions are logged and fed back into critic training, so the critic is always trained on the region refinement pushes into. This closes the self-improvement loop of Q-planning. We also build an anytime scheduler that picks $(N, K)$ online. It uses the measured per-step costs and the spread of critic values among the first few samples, which serves as a cheap proxy for prior entropy.
Relation to Prior Work
Three lines of work motivate the gap. Q-planning and APC-RL show that value signals from sparse rewards can lift behavior priors past their demonstrations. APC-RL does this by routing control between normalizing-flow priors and a prior-free SAC actor with a Boltzmann Q-selector. Both methods leave actions unconstrained, and neither optimizes candidates locally.
Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL shows that sparse-reward RL can beat its sampling-MPC teacher, with policies up to 50% faster and 11–45% lower variance in task duration. It then discards all test-time optimization.
Three model-based lines supply method or constraints, and all require a dynamics model, which we do not.
- Dream-MPC warm-starts gradient MPC from a policy inside a learned world model, at roughly 18 ms on Acrobot.
- MBOP uses a behavior-cloning prior to steer particle MPC and runs at 40–100 Hz.
- DMPC puts a learned value function in the MPC cost to handle sparse-reward navigation.
DiffTORI refines Diffusion Policy actions with a differentiable trajectory optimizer, but without a millisecond-level latency budget or a sparse-reward critic.
One line acts as a constraint on our design. Warm-starting local solvers is known to trap them in the basin of the initialization (Deep Learning Warm Starts for Trajectory Optimization on the International Space Station). This is exactly why we refine several samples from different modes, rather than one warm start, and select only afterward.
Baselines
The naive reference is the frozen Diffusion Policy with STEP acceleration, executed directly. It shows how much any value guidance adds.
The strongest existing competitor is Q-planning sample-and-weight, with the same base policy, the same critic architecture, the same rollout data, and the same number of self-improvement iterations. Its $N$ is tuned to fill the identical wall-clock deadline on the same GPU. This is the right opponent because it is the published method that most directly breaks the imitation ceiling under sparse reward. Its strongest reported numbers are LIBERO-10 93%→99%, RoboTwin 83.8%→91.4%, and real-robot stack-cups 40%→90% and insert-wallet 25%→80% (Beyond Imitation). Beating it means higher final success or shorter episode duration at equal latency on Robomimic Transport and ToolHang and on RoboTwin, plus near-zero constraint violations where Q-planning shows nonzero violations on obstacle variants.
Filtered SFT, which fine-tunes the policy on its own successful rollouts, is run with the same rollout budget. Beating it means exceeding its plateau, which stalled at 55% and 30% on the two real-robot tasks in the same paper.
A model-based reference is TD-MPC2 with Dream-MPC planning, trained on the same data. It tests whether a world model is worth its cost here. We will treat a result where it wins at equal latency as a real negative.
The ablations isolate each ingredient. Setting $K=0$ with argmax selection separates selection from refinement. Averaging the refined candidates instead of selecting tests mode mixing. Removing the trust region tests critic exploitation. Removing the QP isolates the constraint contribution. A fixed $(N, K)$ against the anytime scheduler tests the scheduler.
Why not just…?
Why not just draw more candidates in Q-planning? This is the most serious threat. If the prior already places mass near good actions, larger $N$ may match refinement at the same cost, especially since STEP makes sampling cheap. Our main sweep compares all $(N, K)$ pairs on one deadline curve, so this question is settled directly. If the best $K$ is zero across tasks, the refinement claim fails, and the contribution shrinks to selection plus projection.
Why not just fine-tune the base policy with RL or on its own successes? Filtered SFT stalled well below value-guided ranking in Beyond Imitation, at 55% versus 90% on stack-cups. Sparse-reward RL fine-tuning of large policies is reported to collapse or forget (Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning). A distilled feedforward policy also cannot take new obstacles at test time, as MBOP notes. We still include Filtered SFT as a baseline.
Why not just use full MPC in a learned world model? Dream-MPC already reaches roughly 18 ms, so latency alone does not rule it out. Its weakness is that visual manipulation world models are hard to learn and easy for the optimizer to exploit, and Dream-MPC was evaluated mainly on dense-reward control. The TD-MPC2/Dream-MPC baseline answers this empirically on sparse-reward manipulation.
Why not just replace averaging with argmax? This simple fix may capture most of the safety benefit. It is ablation $K=0$ with argmax. If argmax plus the QP closes the violation gap, we will report that the projection, not refinement, carries the safety result.
Proposed Experiments
Main benchmarks are Robomimic (Lift, Can, Square, Transport, ToolHang, with the proficient-human and multi-human datasets), RoboTwin, and RLBench sparse tasks. For RLBench, STIR² reports about 84% on Slide Block (STIR²) as a model-free reference. For context on Robomimic, BC-RNN reaches 84% on Square and 71.3% on Transport (What Matters in Learning from Offline Human Demonstrations). We avoid LIBERO-10 as a primary test because 99% leaves no headroom.
We also create constrained variants of Transport and Square. These insert unseen box obstacles between demonstrated left and right approach routes and tighten velocity limits. No reported result was found on Lacuna for any method on such variants, so all baselines are run by us.
Metrics are success rate, episode duration relative to demonstrations, constraint-violation rate, and per-step latency at deadlines of 20, 40, and 80 ms.
Three checks decide the claim. First, we measure how far executed actions move from the prior. If the success gains come only from candidates with high prior likelihood, refinement is not extending the support. Second, for refined versus unrefined candidates, we compare the critic's predicted success against the realized success. A growing gap means refinement is exploiting the critic, and the trust region must shrink. Third, PointMaze with wrong-goal demonstrations is a deliberate boundary test, where APC-RL reaches 100% and imitation baselines reach 0–7% (APC-RL). We expect our local refinement to fail there, which would show how far "near the prior" reaches and where a prior-free actor remains necessary.
References
- Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- STEP: Warm-Started Visuomotor Policies with Spatiotemporal Consistency Prediction
- End-to-End Learning to Warm-Start for Real-Time Quadratic Optimization
- APC-RL: Exceeding data-driven behavior priors with adaptive policy composition
- Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL
- Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination
- Model-Based Offline Planning
- Deep Value Model Predictive Control
- DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning
- Deep Learning Warm Starts for Trajectory Optimization on the International Space Station
- Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning
- Reward Relabelling for Combined Reinforcement and Imitation Learning on Sparse-reward Tasks
- What Matters in Learning from Offline Human Demonstrations for Robot Manipulation