Which Simulated Demonstrations Can a Domain-Randomized Robot Policy Afford to Lose?

Robot policies trained only in simulation can work on real hardware if, during training, the simulator keeps changing its physics and appearance. Friction, masses, lighting, textures and camera pose are re-drawn every episode, so the real world looks like one more random draw. This trick, called domain randomization, gives a policy no-real-data ("zero-shot") transfer. It also produces very large demonstration sets, and training a modern vision-language-action model (a large network that maps camera images and an instruction to motor commands) on them is expensive. A separate line of work shows that much robot training data is redundant. On the LIBERO manipulation benchmark, keeping a random 30% of demonstrations matches or beats using all of them. That result was obtained without any randomization. Under domain randomization, the data that matters for transfer is the rare episode where the physics differs and the expert had to act differently. Random thinning may silently throw those episodes away.

We propose a cheap selection rule for domain-randomized demonstration sets. It measures redundancy in a space where each randomized physical parameter is stretched by how much it changes the expert's actions. It then picks a covering subset in that space. Parameters that change behavior, such as friction during a push, get spread-out coverage. Parameters that should be ignored, such as texture, are left to chance, which keeps the visual diversity the policy needs to learn to ignore them. The note also tests the uncomfortable alternative directly: random subsampling may be enough here too.

Core Claim

Our claim is about where the damage from pruning shows up. Under domain randomization, the harm from aggressive pruning appears in the worst-covered regions of parameter space, not in average success. So average success on nominal data, the metric used by Less is More: On Data Redundancy in VLA Training, cannot detect it.

Theory supports this concern. Understanding Domain Randomization for Sim-to-real Transfer treats the randomized simulator as a family of environments with a hidden parameter. It shows that the policy must use its history to infer that parameter, and that the transfer gap depends on how well the training environments separate. A subset that keeps 30% of episodes at random keeps 30% of each parameter region only in expectation. At a 10% budget, the sparse corners of a 10- to 15-dimensional parameter space will empty out.

The selection rule works as follows. Each episode is described by a pooled embedding of its task state, taken from the frozen encoder of the pretrained policy, together with its randomization parameters $\xi$. Each parameter dimension $d$ is rescaled by a sensitivity weight $w_d$. This weight is the fraction of variance in the expert's actions explained by $\xi_d$, conditioned on the task state, and is estimated by a cheap regression on the generated data itself. We then greedily maximize the facility-location coverage function

$$F(S) = \sum_{i \in D} \max_{j \in S} k(z_i, z_j),$$

where $D$ is the full episode set, $S$ is the selected subset, $z_i$ is episode $i$'s sensitivity-weighted feature, and $k$ is a Gaussian similarity. Coverage has diminishing returns: adding an episode near ones already chosen adds little. This makes greedy selection near-optimal.

The rule needs no preliminary training run, unlike dynamic-uncertainty pruning (Large-scale Dataset Pruning with Dynamic Uncertainty) or gradient-variance scoring (Influence Scores at Scale for Efficient Language Data Sampling). It also never targets "hard" frames, the design that failed in Less is More.

Relation to Prior Work

Two papers motivate the gap. Less is More found that random downsampling to 30% matched full-data training for GR00T N1.5 and SmolVLA. It also found that a kinematic difficulty score, the Temporal Surprise Score, did worse than random, which the authors attribute to holes in state-action coverage. That negative result constrains us: we select for coverage, never for difficulty. Every experiment in that paper used clean simulation with no randomization and no real robot.

Active Domain Randomization argues the opposite of Less is More. It holds that uniform sampling of environments wastes effort on configurations already mastered. It addresses this online, with a discriminator and a learned parameter sampler, during reinforcement learning on low-dimensional parameter spaces. Learning Domain Randomization Distributions for Transfer of Locomotion Policies learns a parameter distribution instead. Both shape what is generated. Neither prunes a fixed demonstration set for a large imitation-learned policy, which is our setting.

Distilled Domain Randomization distills per-domain teachers but picks its domains by hand. Our selection could choose those domains, but we do not pursue that here.

Several works supply the method. The coverage objective and its lazy-greedy optimizer come from apricot. The link between nearest-neighbor likelihood and facility location is from Submodularity in Data Subset Selection and Active Learning, which justifies coverage as the right objective for a policy that interpolates between nearby demonstrations. Selection runs at the episode level over tens of thousands of episodes rather than millions of frames. This sidesteps the quadratic similarity cost that apricot flags.

Research Program

We build two domain-randomized demonstration sets that stress different halves of the problem.

The first uses LIBERO (LIBERO) with visual randomization only. We replay the recorded teleoperated state trajectories and re-render them under random textures, lighting, distractors and camera poses. Because actions are unchanged by re-rendering, every visual sensitivity weight should come out near zero. The method should then reduce to task-state coverage with random visuals. This is a built-in sanity check that the sensitivity estimate behaves.

The second uses ManiSkill2 (ManiSkill2) with both dynamics randomization (friction, object mass, damping, actuator latency) and visual randomization. Demonstrations are produced by scripted or motion-planned experts that re-solve each task under each parameter draw, so the actions genuinely respond to the physics. The planned ManiSkill2 parameter set includes PickCube, which has a reported real-robot transfer.

On both sets we fine-tune GR00T N1.5 and SmolVLA at budgets of 5%, 10%, 30% and 100%.

We evaluate on three kinds of test. The first is held-out randomization draws, scored by both mean success and worst-cell success. Worst-cell success is the lowest success over a fixed grid of parameter-space cells. The second is extrapolated parameter ranges. The third is a real-to-sim validated benchmark: we use REALM (REALM), whose 14 calibrated Franka parameters double as randomization axes. Its tiered Reach-Grasp-Lift-Move progression score correlates with real-world progression at r ≈ 0.921. If a physical arm is available, we also run PickCube on hardware.

Baselines

The naive reference is training on 100% of the randomized demonstrations. This is the configuration every domain-randomization paper in the file implicitly uses. Beating it means matching its mean success at a 10–30% budget while not losing worst-cell success.

The strongest competitor is uniform random subsampling. Less is More reports GR00T N1.5 at 81.33% LIBERO success with 30% random data, versus 80.33% with full data. SmolVLA reached 60.50% versus 59.75%. Performance was stable until roughly 5–10% of data. We expect random subsampling to stay close to full data on mean success under randomization as well. Our method beats it only if, at equal budget, it raises worst-cell or extrapolation success by more than the seed-to-seed spread over at least three seeds, without lowering mean success.

Stratified random sampling over parameter bins is the next most natural competitor. It guarantees per-region coverage without any learned weighting. We give it the same bin grid used for scoring. This is slightly generous to it, because the bins match the evaluation cells.

The key ablation is facility location on unweighted parameters ($w_d = 1$ for all $d$). It isolates whether the sensitivity weighting matters or whether plain coverage does the work.

We include two further comparisons. The Temporal Surprise Score represents difficulty-based selection; Less is More reports 74.00% for GR00T N1.5. Dynamic-uncertainty pruning represents training-dynamics scoring. It is charged for its scoring run, so its total compute is reported alongside its accuracy.

All methods share the same generated episode pool, the same fine-tuning recipe and step count, and the same hyperparameter search budget, so differences come only from which episodes are kept.

Why not just…?

Why not just subsample at random, since that already matched full data on LIBERO? This is the real threat. The evidence for random subsampling comes entirely from un-randomized LIBERO and from mean success. Nothing in the file tests it on tail parameter regions or on transfer. If random sampling matches our method on worst-cell success and on REALM progression at 10% and 30%, the correct conclusion is that redundancy under domain randomization is benign. We will report that as the finding rather than defend the method.

Why not just stratify by parameter bins? For two or three randomized parameters, stratification is probably as good as anything. With 14 REALM parameters plus visual factors, a full grid has far more cells than episodes. Uniform stratification also spends budget spreading coverage over parameters like texture hue that need no coverage, only randomness. The comparison against both stratification and the unweighted-coverage ablation tells us whether sensitivity weighting adds anything beyond coverage.

Why not just generate fewer, better environments with Active Domain Randomization or a learned distribution? These shape data at generation time and are complementary to our approach. Active Domain Randomization adds a discriminator and a learned sampler with reported instability. It was demonstrated on spaces of at most about eight parameters with reinforcement learning, not on demonstration sets for large pretrained policies. Generation-time and selection-time curation can be stacked. We treat this pairing as follow-up work rather than a competitor in this study.

Why not just train on everything, since simulation is cheap? For large pretrained policies, simulation is not the expensive part; fine-tuning is. ManiSkill2's simulator generates samples at 2,532 FPS, while fine-tuning GR00T-scale models is the bottleneck. The full-data baseline also directly measures whether extra data helps or hurts transfer. The file's evidence that redundant data can hurt comes from language and regression tasks. This experiment checks whether that holds for robots.

Decisive Checks

The central measurement is the gap between mean success and worst-cell success as the budget shrinks.

If random subsampling keeps mean success but loses worst-cell success at 10%, and the sensitivity-weighted subset keeps both, the core claim holds. The REALM tiered progression score should move in the same direction, so the effect is not an artifact of our own grid.

Two outcomes would falsify the claim. One is that random subsampling holds worst-cell success. The other is that unweighted coverage does as well as the weighted version.

A third check validates the mechanism. Measured sensitivity weights should be near zero for the LIBERO visual-only set. On ManiSkill2 they should be large for friction and mass, the parameters that change what the expert does.

The physical test is PickCube on hardware. The file reports 91% success in simulation and 60% on the real robot without fine-tuning, but that was the original ManiSkill2 pipeline, not our full-data fine-tuning setup. So we re-measure the full-data success ourselves on hardware. The question is whether a 10% sensitivity-weighted subset keeps that re-measured full-data success while a 10% random subset does not.