Robots trained in simulation are made robust to the real world by domain randomization: each training episode draws new physical constants (friction, object mass, joint damping, motor strength), so the policy has to work across all of them. Which constants to vary, and over what ranges, is still chosen by an expert. Set the ranges too narrow and the policy breaks on the real robot. Set them too wide and training wastes effort on physically impossible settings, and the policy becomes timid. Recent work has removed much of the expert effort elsewhere in the pipeline. Rewards can be reduced to a binary success signal, and task curricula can be generated automatically. The randomization ranges are one of the last hand-tuned pieces, and they are often decisive for transfer.
We propose to replace the expert's ranges with a signal computed from binary success alone. For a candidate physics setting, we briefly fine-tune a copy of the main policy on that setting alone, producing a "specialist." We then measure how much more often the specialist succeeds than the general policy. This gap is zero for settings that are impossible (nobody succeeds) and zero for settings that are already mastered (everyone succeeds). It is positive exactly where the general policy is failing on something that is physically achievable. Sampling training physics in proportion to this gap should let us start from deliberately loose, physically plausible bounds and still match or beat carefully tuned ranges when policies are transferred zero-shot to real hardware.
Core Claim
The claim is narrow and testable. With binary success rewards, "specialist regret" is a feasibility-aware curriculum over physics parameters. It needs neither a hand-set feasibility threshold nor hand-tuned ranges. Its specific advantage over simpler success-based filters is that it trains on the coverage holes: settings where the general policy's success is zero even though a specialist can succeed.
Write $\xi$ for a vector of physics parameters. Write $\hat p_G(\xi)$ for the general policy's empirical success rate on $\xi$, and $\hat p_S(\xi)$ for the success rate of a specialist obtained by fine-tuning the general policy for a fixed short budget on $\xi$ alone. The curriculum score is
$$r(\xi) = \max\big(0,\ \hat p_S(\xi) - \hat p_G(\xi)\big).$$
Training physics are drawn from a buffer of candidate settings with probability increasing in $r(\xi)$, mixed with a small share of uniform samples from the loose bounds.
How This Relates to Prior Work
The regret structure comes from PAIRED (Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design). PAIRED rewards an environment generator by the performance gap between an "antagonist" policy and the learner. That gap is zero for impossible and for trivial environments, and PAIRED reports 40% zero-shot success on a Labyrinth maze against 10% for a minimax adversary and 0% for uniform randomization. PAIRED supplies our method, but its antagonist is a second generalist trained on shaped rewards, and it was demonstrated on mazes and a Hopper under perturbations. We make the antagonist a per-setting specialist, restrict rewards to binary success, and target physics ranges for real transfer.
Feasibility is the constraint we must respect. Feasible Adversarial Robust Reinforcement Learning for Underspecified Environments shows that unconstrained adversaries push training into impossible configurations. It fixes this with a threshold $\lambda$ on a hand-crafted reward and a penalty $C$, both chosen by the engineer. Under binary success, "some policy can succeed" becomes a statistical question about whether the specialist succeeds at least once. The only remaining knob is the number of probe episodes, which controls confidence and carries no task-specific knowledge.
Two lines of work motivate the gap. Active Domain Randomization adapts physics parameters by rewarding a particle-based generator when a discriminator can distinguish rollouts from a reference environment. Generating Automatic Curricula via Self-Supervised Active Domain Randomization pairs that generator with self-play goal setting on low-DoF Poppy Ergo Jr arms. In both methods, "the behavior differs" is the signal, and that signal cannot distinguish a hard-but-possible setting from an impossible one. Both also still rely on expert-set parameter bounds.
DemoStart supplies both the pipeline we build on and the simple competitor we must beat. It learns dexterous skills from binary rewards and a few demonstrations, trains only on task parameters with nonzero success variance ("zero-variance filtering"), and distills to a vision policy under hand-tuned uniform randomization. It reaches 97% real success on plug lifting and cube reorientation, but only 64% on plug insertion. Asymmetric Actor Critic for Image-Based Robot Learning supplies the privileged-critic training we reuse. It gives the critic full simulator state while the actor sees only images, trains with sparse rewards and hindsight relabeling, and transfers on 5/5 real trials with uniform visual randomization.
LLM-driven generators address different expert knobs. Environment Curriculum Generation via Large Language Models co-evolves terrain code with a policy. CurricuLLM writes subtask rewards. Neither sets physics distributions, and both still depend on fixed or generated shaped rewards.
Research Program
We keep everything else fixed so that the randomization ranges are the only expert input being replaced. The task side uses binary success, and on dexterous tasks it uses DemoStart-style demonstration seeding. The loose bounds are set once per parameter from generic physical plausibility, for example friction over roughly two orders of magnitude or mass from a fifth to five times nominal, with no per-task tuning. We record the time spent choosing these bounds, as a proxy for residual expert effort.
The algorithm has three parts. First, a candidate buffer is filled with uniform draws and with small perturbations of high-scoring settings. Second, a probe step estimates $\hat p_G$ and $\hat p_S$ on selected candidates. Specialists are fine-tuned from the current general policy for a fixed small budget and then discarded. The probe budget is re-estimated only for candidates whose score is stale. Third, the general policy trains on a regret-weighted mixture of settings. All probe environment steps count against the method's total simulation budget. That budget accounting is the main engineering risk, and we treat it as part of the result.
The analysis has two parts. We measure the coverage hole: the fraction of settings, among those where a well-funded specialist succeeds, on which the final general policy never succeeds. We also track how the effective sampling distribution evolves relative to the expert's hand-tuned ranges. This shows whether the method rediscovers the expert's ranges, exceeds them, or collapses onto a narrow easy region.
Baselines
The naive reference is uniform randomization over the same loose bounds, trained with the same sparse-reward learner and the same total simulation steps. This measures what loose bounds cost when nothing selects among them.
The strongest existing reference is uniform randomization over expert-tuned ranges. On dexterous tasks we use DemoStart's recipe with its reported ranges where available. On Poppy Ergo Jr tasks we use the ranges from the Active Domain Randomization papers. On the NIST-style dexterous suite the numbers to match are DemoStart's reported real-robot success rates of 97% (plug lifting), 64% (plug insertion) and 97% (cube reorientation). Plug insertion is the task with room to improve. For Poppy Ergo Jr we found no numeric real-robot result on Lacuna for ADR or SS-ADR; the file reports only qualitative claims ("tight spread", "lowest average final distance"), so we rerun those baselines ourselves. We found no reported numbers on Lacuna for robosuite tasks either. Beating this reference means matching its real success rate without using its ranges, and exceeding it on held-out physics or on plug insertion.
Active Domain Randomization with its discriminator reward is the adaptive competitor. We run it over the loose bounds with an equal step budget, to test whether a feasibility-blind adversary drifts into impossible settings. FARR-style feasibility with $\lambda$ and $C$ tuned by grid search on a validation set shows what threshold-based feasibility achieves when it is given extra tuning.
The key ablation is zero-variance filtering over physics parameters: train only on settings where the general policy's success variance is nonzero, with no specialists. This removes our one new ingredient. A second ablation replaces the per-setting specialist with a PAIRED-style generalist antagonist, to test whether specialization matters. All methods receive the same network, learner, seeds (at least five), tuning budget for their own hyperparameters, and total simulation steps with probes included.
Why not just…?
Why not just use the expert's ranges? They work, and DemoStart's 97% real success on two tasks shows it. The point is to remove that tuning step, not to argue that it fails. The honest risk is that our method only matches the expert. That is still a positive result if it holds across tasks where no one tuned anything. The 64% plug-insertion result suggests hand-tuned ranges may leave contact-sensitive regimes uncovered, and the held-out physics tests are designed to measure exactly that.
Why not just randomize widely and let training sort it out? The evidence against this comes from neighboring settings. PAIRED reports 0% transfer for uniform randomization on its Labyrinth. Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs finds that uniform sampling yields solvable pairs only 2.7% of the time in rich spaces, and that randomization fails to converge there. Neither result is about physics ranges for manipulation, so our loose-bounds baseline tests the question directly.
Why not just apply zero-variance filtering to physics? This is the real threat. Impossible settings never produce success, so the filter removes them automatically and needs no specialists, which makes it far cheaper. Its blind spot is the one we target. A setting where the general policy always fails also has zero variance, so the filter discards feasible-but-unlearned settings as well as impossible ones. The policy can expand into those settings only if neighboring settings happen to be learned first. If the coverage-hole measurement and the held-out success show no difference between the two methods, we will report that filtering suffices and that specialists are not worth their cost. That would be a clean and useful negative result.
Why not have an LLM propose the ranges? Eurekaverse and CurricuLLM show that language models can write environment and reward code. However, they work from text summaries of rollouts, and the research file notes they lack grounding in contact dynamics. We include one LLM-proposed-ranges run as a secondary comparison, but we do not expect it to resolve feasibility near contact boundaries.
Proposed Experiments
Stage one runs in simulation on ErgoReacher and ErgoPusher and on robosuite Lift, Door and Nut Assembly with a Panda arm under operational-space control, all with binary success. To stand in for the reality gap, each task gets a held-out "hidden world" set of physics settings that no method sees. Half are drawn inside the expert ranges and half outside them but within the specialist-feasible region. The metrics are held-out success, nominal-setting success (a test of timidity), coverage hole, and simulation steps to reach a target success, with probes counted.
Stage two transfers the best and baseline policies zero-shot to real Poppy Ergo Jr arms, using final distance and success over 20 trials per condition. If hardware allows, it also runs a multi-fingered plug-insertion setup that follows DemoStart's distillation pipeline, where the randomization ranges are the only change.
Two outcomes are decisive. First, specialist regret over loose bounds should match expert-range randomization within confidence intervals on real hardware. Second, it should beat zero-variance filtering on the outside-range held-out set, with a measurably smaller coverage hole. If the first outcome fails, the method does not remove the expert. If the second fails, the specialists are unnecessary and the simpler filter should be recommended instead.
References
- Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
- Feasible Adversarial Robust Reinforcement Learning for Underspecified Environments
- Active Domain Randomization
- Generating Automatic Curricula via Self-Supervised Active Domain Randomization
- DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robots
- Asymmetric Actor Critic for Image-Based Robot Learning
- Environment Curriculum Generation via Large Language Models
- CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Models
- Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs