Searching for Recovery Data: Student-Targeted Population Search in GPU Simulators

Robot imitation policies fail in a predictable way. A small execution error puts the robot in a state the demonstrations never visited, and the policy has nothing to copy. Systems that multiply a few human demonstrations into thousands, such as MimicGen, replay transformed copies of the human motion from nominal starting states. That produces many near-copies of what the human already did, and few demonstrations of how to get out of trouble. The symptom is visible at scale: RoboCasa generated over 100,000 such trajectories, yet composite long-horizon tasks stayed below 12% success.

We propose to spend GPU-parallel simulation on a different question: not "what else could the human have done?" but "from the exact states where the current student policy goes wrong, what short trajectory recovers and finishes the task?" The method has three steps. A diffusion policy, meaning an imitation model that generates action sequences by gradually denoising random noise (Diffusion Policy), is trained on a handful of human demonstrations. Its failure states are then harvested from batched rollouts in ManiSkill3. From each failure state, a population of thousands of candidate edits to the human demonstration's remaining segments is evaluated in parallel, and the edits that succeed are kept. The result works like DAgger, the imitation scheme that queries an expert at the states the learner actually visits, except that the expert is replaced by population search over human-anchored trajectories. The search loop stays entirely on the GPU, in the style of evosax and QDax.

Core Claim

Given a fixed simulation budget, data generated by searching from the student's own failure states will raise downstream success on multi-stage manipulation more than the same budget spent on uniform replay-based generation. The gain should come mostly from states where rigid segment replay cannot find a solution, such as a half-grasped object, an object knocked out of pose, or a gripper misaligned with an insertion target. In those states, a transformed copy of the human segment usually fails, and only a search over its geometry and timing finds a recovery. Two things would make the work new, and both are testable separately. The first is targeting: the data distribution follows the student's failures. The second is search: populations of edits are evaluated in parallel, rather than replaying a single edit and filtering it.

Relation to Prior Work

Three lines of work motivate the gap. MimicGen parses roughly 200 human demonstrations into object-centric segments, re-poses them for new object placements, and filters by task success. It generated over 50,000 demonstrations, but it reproduces the demonstrator's strategy and does not search. RoboCasa scales that pipeline and exposes its limit on composite tasks. What Matters in Learning from Offline Human Demonstrations shows that offline reinforcement learning on human data collapses to 0–10% success. This constrains us: we should not try to extract improvement from human data by fitting a value function to it. Instead we generate new, physically verified supervision.

Three works come closest to the proposal. DexFlyWheel grows one demonstration into more than 2,000 by training a residual reinforcement-learning correction on top of an imitation policy and keeping successful rollouts. It reports 81.9% versus 45.2% for DexMimicGen. However, it needs hand-shaped rewards, it accepts any success, and it does not target the student's failures. Quality Diversity Imitation Learning combines MAP-Elites-style archives (grids of solutions indexed by a behavior description, each cell keeping its best solution) with adversarial imitation from 4 demonstrations. Its output is an archive of locomotion policies, not a training corpus, and it does not run on GPU. Genetic Diffusion Policy evaluates populations of candidate action chunks inside GPU batches, but only at inference time on a frozen policy. It shows that batched population selection is cheap enough to use, while leaving the training data untouched.

The method draws its machinery from four sources. evosax and QDax supply on-device search and archives. GMPEA supplies the practice of replacing branching constraint checks with tensor masks; its 62.6x speedup on constrained problems is the reason we write constraints this way. ManiSkill3 supplies heterogeneous batched physics and rendering.

Research Program

Trajectory parameterization. A demonstration's remaining segments are described by a low-dimensional edit vector $\theta$. It contains object-frame offsets to a few waypoints, a grasp-pose delta, a per-segment time scaling, and two to four spline knots for the transition from the failure state into the segment. This gives roughly 20–60 dimensions. Keeping it small is deliberate. evosax reports that evolution strategies degrade beyond about $10^5$ parameters, so we search over edits to human motion, never over policy weights.

Fitness. For a candidate starting from failure state $s$, fitness is

$$F(\theta; s) = \mathbb{1}[\text{success}] + \lambda \, \frac{k(\theta; s)}{K} - \mu \, \|c(\theta; s)\|_1 .$$

Here $k$ is the number of the task's $K$ subtask predicates completed. These are the same subtask boundaries MimicGen already uses. $c$ is a vector of masked constraint violations: penetration depth, joint-limit excess, and contact-force excess. $\lambda$ and $\mu$ are fixed weights. No task-specific reward shaping is added; this is the contrast with DexFlyWheel.

Archive. Each cell is indexed by a failure-state cluster crossed with a coarse approach direction for the recovery. Failure states are clustered on object-relative gripper pose and object pose error. Indexing by approach direction keeps more than one recovery strategy per failure mode, which is the only place quality-diversity enters. The emitter is a CMA-ES or SNES (two standard evolution strategies) instance per cell, warm-started at $\theta = 0$, which is plain MimicGen replay. Search can therefore only add to replay.

Loop. Each round has four steps. Train the diffusion policy on the current data. Roll it out in $10^3$–$10^4$ parallel environments with randomized initial states and mid-episode perturbations. Snapshot simulator state at divergence points, defined as the first step where a subtask predicate regresses or progress stalls. Run search from those snapshots and append the successful archive elites, rendered as observation–action pairs, to the data. Snapshots are restored directly into batched environments, so search never leaves the GPU.

Baselines

The naive reference point is a diffusion policy trained only on the seed human demonstrations. We use 5 and 10 per task, which is the regime where Diffusion Policy is known to degrade. Every other method must clearly beat this floor.

The strongest existing method is MimicGen-style replay generation, reimplemented in ManiSkill3 and given the same total simulation steps as our method. The best reported figures for this family come from RoboCasa: adding 3,000 MimicGen demonstrations to 50 human ones raised atomic-task success from 28.8% to 47.6%, while composite tasks stayed below 12%. We found no reported result on Lacuna for MimicGen on ManiSkill3 tasks, so we will establish that number ourselves. Beating it means a statistically reliable gain in downstream success at equal simulation steps and an equal tuning budget for the student, both on held-out initial states and under mid-episode perturbation.

The second strong competitor is a DexFlyWheel-style residual reinforcement-learning flywheel, run in the same simulator. It gets the same seed demonstrations and the same steps, with the same subtask-predicate reward as our fitness, so it is not handicapped by a weaker reward. DexFlyWheel's reported 81.9% versus 45.2% for DexMimicGen in OmniGibson shows that this family can beat replay by a wide margin.

Two ablations isolate the key ingredients. The first is failure-state replay: MimicGen restarted from the student's harvested failure states, with best-of-$N$ filtering over random perturbations of $\theta$ at the same number of simulated evaluations, but with no search update. This separates targeting from search. The second is untargeted search: the same evolutionary archive, run from uniformly sampled initial and perturbed states rather than student failures. This separates search from targeting.

Why not just…?

Why not just generate far more MimicGen data? RoboCasa is the direct evidence that volume alone does not fix composite tasks: 100,000+ trajectories still left them below 12%. Our matched-budget comparison tests this directly. If uniform replay with our full simulation budget matches us, the claim fails.

Why not just restart MimicGen from the student's failure states? This is the real threat. It may capture most of the benefit with no search at all, since MimicGen already re-poses segments relative to objects. The failure-state replay ablation settles it. We will report the yield, meaning the fraction of failure states with at least one successful trajectory, for replay alone and for search. If downstream success is equal, the honest conclusion is that targeting is the contribution and the evolutionary machinery is unnecessary. We would publish the work in that form.

Why not just use residual reinforcement learning, as DexFlyWheel does? Residual reinforcement learning optimizes one policy toward reward. It tends to collapse to a single recovery strategy and depends on reward shaping. Population search over human-anchored edits needs no gradient through contact, and every elite remains a bounded edit of human motion. Whether that matters empirically is the head-to-head comparison above. We expect residual reinforcement learning to be competitive on single-stage dexterous tasks and weaker on multi-stage ones.

Why not just use gradient-based reannotation, as in Learning to Drive Anywhere with Model-Based Reannotation? That method relabels data by optimizing through a learned differentiable dynamics model, and it is shown only for planar navigation. Gradients through grasp and insertion contacts are unreliable. We include a CEM/gradient-free versus differentiable-simulator comparison only if a differentiable path through ManiSkill3 contact is available. Otherwise we note this as untested.

Proposed Experiments

The main suite uses multi-stage rigid manipulation from ManiSkill3's task categories, namely stacking, insertion, and pick-and-place with articulated objects. We use 5 or 10 seed demonstrations collected through its teleoperation interface. Each method gets a fixed budget of simulator steps per task, and every student is the same diffusion-policy architecture with identical training and tuning budgets. Evaluation covers three conditions: success on held-out initial states, success under scripted mid-episode perturbations (object pushes and grasp slips), and success as a function of simulation budget. We also report generation yield per GPU-hour and archive coverage of failure clusters.

The decisive checks follow from the claim. Targeted search should beat matched-budget uniform replay. Its advantage over failure-state replay should be concentrated in failure clusters where replay yield is low. Gains should grow with task horizon, which we test by comparing two-stage against four-stage variants. A secondary check runs the best-found pipeline on Adroit or Robomimic tasks through a CPU-batched simulator, purely to confirm that the gains are not a ManiSkill3 artifact. We accept the wall-clock cost there and make no throughput claim.

The project is falsified if either of two outcomes holds. One is that failure-state replay matches targeted search at equal evaluations; then search adds nothing. The other is that untargeted search matches targeted search; then student failures are not the right place to look.