Robots that plan with a numerical trajectory optimizer, a solver that iteratively improves a candidate path until it is low-cost and satisfies the physics and safety constraints, are often too slow for real-time control. The standard fix is a learned "warm start": a neural network guesses a good initial path so the solver needs few iterations. Today these warm-starters are trained to copy the outputs of a perfect solver. The guess is only as good as its training targets, and a bad guess can leave the solver slower than starting from scratch. Real data for a new robot or task often comes instead from humans or rough scripts, so it contains fumbles, detours, and conflicting strategies. Copying that data averages incompatible paths into initial guesses the solver cannot use.
We propose to stop treating demonstrations as targets and treat them as raw material for the solver. Each demonstrated state becomes a trajectory optimization problem. The training label for that state is whatever the solver converges to from the best available initialization. Those labels are produced by a backward sweep along each demonstration, starting near the goal, the way a teacher designs a course backward from the final competency. A reverse curriculum normally requires resetting a robot or simulator to arbitrary states. Here it costs nothing, because "resetting" an optimization problem only means changing its initial condition. In two sentences: demonstrations are graded by what the solver makes of them, not by how closely a policy can copy them. The grading runs backward from the goal, so each newly labeled state can reuse the already-solved, near-optimal remainder of the path.
Core Claim
A warm-starter trained on solver-converged labels from a backward, outcome-graded sweep over imperfect demonstrations has three properties. It should match the iteration savings of a warm-starter trained on clean solver data. It should stay below cold-start iteration counts even when most demonstrations are corrupted. Its labels should have lower cost than the demonstrations themselves, so it is not bounded by the best path in the data. The mechanism rests on one observation about nonconvex trajectory optimization. A demonstration does not need to be optimal to be useful. It needs to place the solver in the right basin, for example on the correct side of an obstacle. Return-based quality scores do not measure that property. Solver outcomes do.
Relation to Prior Work
The proposal draws on three lines of work, each in a different role.
The latency literature supplies the method and the main failure we target. Deep Learning Warm Starts for Trajectory Optimization on the International Space Station trains a small network on about 11,000 solver-generated trajectories to warm-start the GuSTO sequential convex programming solver on the Astrobee robot. It cuts iterations by 60% on rotations and 50% on obstacle avoidance. It also reports that, on unseen obstacle layouts, the learned guess needed more iterations than a cold start. That is the failure mode imperfect training data would make routine. End-to-End Learning to Warm-Start for Real-Time Quadratic Optimization avoids labels entirely by differentiating through Douglas-Rachford splitting iterations on quadratic programs. Unrolling a nonconvex solver over long horizons is much more expensive, so we generate labels by running the solver forward rather than by backpropagating through it. STEP warm-starts a diffusion policy with a learned predictor and reaches 20 ms latency, but it assumes the underlying policy was trained on good demonstrations. DiffTORI embeds a differentiable optimizer in a policy. Its imitation loss pushes the optimizer output toward the demonstrations, so jitter and detours get fitted into the learned cost. We keep the optimizer's cost fixed and change only what the warm-starter learns from.
The suboptimal-demonstration literature motivates the problem and constrains the design. MARWIL weights the imitation loss by exponentiated advantage. Confidence-Aware Imitation Learning and demonstrator expertise estimation infer per-trajectory or per-demonstrator quality. All three still imitate the data they keep, so they stay bounded by the best transitions present. That is the same ceiling the file notes for COIL. IRIS stitches good segments from fumbling crowdsourced data using a value function, but it uses no solver and gives no latency or feasibility guarantee. Our "quality score" is the solver's own converged cost and iteration count. Our labels can be better than any demonstration because the solver improves them.
The curriculum literature supplies the backward-expansion idea and a key limitation we remove. RFCL resets the simulator to demonstration states near the goal and steps backward as the agent succeeds, keeping a separate curriculum per demonstration to avoid mixing strategies. It depends on env.set_state(). BaRC grows backward reachable sets with Hamilton-Jacobi analysis, a method whose cost grows exponentially with state dimension. We keep RFCL's per-demonstration backward ordering. Our resets happen inside the solver's model, never on hardware. Backward progress also needs no reachability computation, because each step reuses a path the solver has already certified as feasible. Neither RFCL nor BaRC produces a fast deployable optimizer, and neither uses demonstrations as solver initializations.
Research Program
The pipeline has three parts.
The first part is the inputs. We take a dynamics model and a fixed trajectory optimization problem: cost, constraints, and goal set. We also take a set of demonstrations of arbitrary quality.
The second part is the backward labeling sweep. For each demonstration $i$ with visited states $s^i_0,\dots,s^i_T$, we walk from $t=T$ toward $t=0$ in strides of $m$ steps. At each state $s^i_t$ we pose the problem "reach the goal from $s^i_t$" and run the solver from a small set of candidate initializations under a fixed iteration budget. There are four candidates:
- The remainder of the demonstration from $s^i_t$.
- A cold start.
- The current warm-starter's prediction.
- A stitched candidate. It follows the demonstration from $s^i_t$ to $s^i_{t+m}$ and then appends the already-converged solution $x^\star(s^i_{t+m})$.
The stitched candidate is where the backward ordering matters. Close to the goal, problems are short and cold starts converge reliably. Further back, the near-optimal tail is already known, so only a short prefix needs repair. The label for $s^i_t$ is the candidate outcome that minimizes
$$ \ell(s) = J(s) + \lambda\, \frac{k(s)}{K}, $$
where $J(s)$ is the converged cost (infinite if the solve is infeasible or does not converge within the budget $K$), $k(s)$ is the number of iterations used, and $\lambda$ trades solution quality against solver effort. Demonstrations are never discarded outright. A fumbling demonstration still contributes its states and its route around obstacles. Its mistakes are erased by the solver rather than copied.
The third part is the warm-starter. It outputs $M$ candidate trajectories, for example as polynomial coefficients as in the Astrobee work. It is trained with a best-of-$M$ regression loss on the labels, so different solution modes can occupy different outputs instead of being averaged. The warm-starter is retrained every few backward strides, which is what makes the curriculum adaptive. Its predictions become better candidates for the next stride, and states where it already matches the label can be skipped. At deployment, the $M$ candidates and a cold start are solved in parallel within the latency budget, and the first one to converge is used. This deployment rule is a guard against the out-of-distribution failure seen on the ISS, and we report results with and without it.
The analysis has three goals. We will measure how label cost and label-generation compute scale with the stride $m$. We will check whether backward stitching actually lowers converged cost relative to solving each demonstration state independently. We will characterize the regime where it does, which we predict is long-horizon nonconvex problems where demonstrations sit far from optimal.
Baselines
The naive reference points are a cold-started solver and a short-horizon cold-started solver. They are run with the same solver, cost, budget, and hardware. Beating them is the minimum bar: fewer iterations at equal or lower converged cost.
The strongest existing method is the Astrobee recipe from Deep Learning Warm Starts for Trajectory Optimization on the International Space Station. It trains the same network architecture on cold-start solver solutions over sampled problems and reports 60% (rotations) and 50% (obstacle avoidance) iteration reductions at the same converged cost as a cold start. It is the strongest competitor because it never touches imperfect data, so it represents the "just ignore the demonstrations" answer. We run it with the same total number of solver calls our sweep consumes, drawing problem instances uniformly from the workspace. On the quadratic-program vehicle task we also run the unrolled method of End-to-End Learning to Warm-Start, which reports 48 iterations against over 600 for a cold start. We expect that method to win there, and that task serves as our convex negative control. To beat the Astrobee recipe, our method needs lower median iterations and a higher fraction of solves inside a 20 ms budget at matched converged cost on held-in layouts. It should also not lose to it on held-out layouts.
The imitation baselines train the same warm-starter directly on raw demonstrations in four ways: unweighted, top-25% filtered by cost, MARWIL-weighted (Exponentially Weighted Imitation Learning for Batched Historical Data), and confidence-weighted (Confidence-Aware Imitation Learning). All use the same tuning budget, here 20 configurations per method on a validation split.
Three ablations isolate the key ingredients:
- "Polish-once" solves every demonstration state independently, starting from its demonstration remainder, and regresses on the result. It removes the backward ordering and stitching.
- Forward sweep keeps stitching but walks from the start state toward the goal.
- Single-output warm-starter uses one output instead of $M$.
Why not just…?
Why not just filter to the best demonstrations and regress on them? Filtering keeps the demonstrator's mistakes inside the trajectories it retains and shrinks coverage. COIL shows the coverage cost for policies: on D4RL hopper-medium, plain BC scored about 923 against about 3012 with curriculum imitation. The ISS result shows that a warm start that is merely close in the wrong way can be worse than none. The demonstration-quality sweep in our experiments measures exactly where filtered regression crosses above cold-start iteration counts.
Why not just generate labels with cold-started solves and skip demonstrations, as on the ISS? This is a real threat. When cold starts reliably find good basins, demonstrations add nothing. Our bet is that in cluttered, long-horizon problems cold starts often land in poor local minima or fail. Demonstrations, even sloppy ones, encode which route around obstacles works. The matched-solver-call comparison settles this. If the cold-start recipe ties us on cluttered layouts, the demonstrations are not earning their keep, and we will report that.
Why not just polish each demonstration with the solver once and regress on the result? This is the most dangerous alternative. It captures most of the "solver grades the data" idea without the curriculum. We expect it to match our method on short horizons and on the convex control task, and to fall behind when demonstrations are far from optimal over long horizons. In that regime, polishing from the raw demonstration can stall at the iteration budget or converge to a poor solution, while the stitched candidate only needs to repair a short prefix. If polish-once matches our method on the long-horizon, heavily corrupted settings, we will drop the backward sweep from the claim. The contribution would then reduce to outcome-graded labels with a multimodal warm-starter.
Why not just use a diffusion policy accelerated with STEP? STEP makes a demonstration-trained policy fast, but it still reproduces the demonstrations and gives no constraint satisfaction. We include a STEP-accelerated diffusion policy trained on the same corrupted data as a closed-loop reference on the manipulation task. We compare constraint violations and success, not solver iterations.
Proposed Experiments
The primary testbed is a 6-DoF free-flyer obstacle-avoidance problem modeled on the Astrobee setup described in the ISS work. Its demonstrations are deliberately imperfect, which gives a controlled corruption axis with known ground truth. We mix cold-start solver solutions with three kinds of corrupted paths: jittered trajectories, deliberate wrong-side detours, and solutions stuck in local minima. We sweep the corrupted fraction from 0% to 95%, following the spirit of TRAIL's "bottom 5% of available quality" stress test. Obstacle layouts are split into held-in and held-out sets.
The second testbed uses D4RL Maze2D and AntMaze (D4RL) with model-based trajectory optimization over a known point-mass model and a learned ensemble model respectively. The D4RL trajectories there are naturally suboptimal and require stitching.
The third testbed is the vehicle-dynamics quadratic program from the end-to-end warm-start paper. It is the convex control, where we predict no benefit from backward ordering.
As an optional demonstration on human data, we use the uncurated RoboTurk Cans and Robosuite Lift demonstrations studied by IRIS. There, an end-effector trajectory optimizer built on the DiffTORI codebase serves as the solver.
The metrics are:
- Solver iterations to convergence.
- Wall-clock time per solve on a fixed GPU.
- Deadline hit rate: the fraction of solves converged within 20 ms, the budget STEP reports for edge deployment.
- Converged cost relative to the best of many random restarts.
- Constraint violation.
- Closed-loop task success.
For reported numbers on the comparison points: the Astrobee warm-starter reports 50–60% iteration reductions on solver-optimal data, and the quadratic-program warm-starter reports 48 versus over 600 iterations. We found no reported result on Lacuna for learned warm-starting of trajectory optimizers from suboptimal or human demonstrations. The comparison under corruption is therefore new and must be run here.
Four outcomes would be decisive:
- At 80% corruption or more, raw-demonstration regression should exceed cold-start iteration counts, reproducing the ISS failure, while our method stays within 10% of the clean-data warm-starter's iterations.
- Our labels should have lower cost than the demonstrations they came from on the majority of states. That would show the method is not capped by the data.
- On long-horizon held-in problems, backward stitching should beat polish-once on converged cost and deadline hit rate. On the convex control task it should not.
- On held-out obstacle layouts, our method with the cold-start fallback should never be slower than a cold start. Without the fallback, we report how often it is.
If outcome 3 fails, the curriculum is not what drives the gains, and the claim narrows to solver-graded labeling.
References
- Deep Learning Warm Starts for Trajectory Optimization on the International Space Station
- End-to-End Learning to Warm-Start for Real-Time Quadratic Optimization
- STEP: Warm-Started Visuomotor Policies with Spatiotemporal Consistency Prediction
- DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning
- Exponentially Weighted Imitation Learning for Batched Historical Data
- Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality
- Imitation Learning by Estimating Expertise of Demonstrators
- Curriculum Offline Imitating Learning
- IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data
- Reverse Forward Curriculum Learning for Extreme Sample and Demo Efficiency
- BaRC: Backward Reachability Curriculum for Robotic Reinforcement Learning
- TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning.