A standard way to train a robot policy that works on hardware without any real-world practice is domain randomization: every simulated episode draws new physics (masses, friction, damping, center of mass), so the policy has to cope with a range of worlds that, one hopes, includes the real one. The catch is well documented. If the range is wide enough to cover reality, plain training often collapses into timid or motionless behavior. On a real Franka arm pushing a box under 17 randomized parameters, a policy trained on a fixed wide distribution succeeded 0% of the time, while an adaptive curriculum that widens the range gradually reached 60% (Domain Randomization via Entropy Maximization). The field's response has been to manage the environment distribution, by widening it slowly, backtracking when performance drops, or adversarially picking hard worlds (Active Domain Randomization).
This proposal asks whether that collapse is really premature convergence in the policy optimizer. Premature convergence means the learner settles on a mediocre solution before finding a better one that exists. If the timid policy is a trap rather than the best achievable compromise, then the right fix is in the optimizer and not in the environment schedule. We propose to test this directly. We then introduce a targeted fix: keep the policy's action noise from shrinking until the policy does about as well as is known to be possible in its worst physics regime. This adapts a goal-achievement exploration floor from robot locomotion so that it is indexed by dynamics regime instead of by reward term.
Core Claim
We make two claims. First, under wide randomization on PandaPush, the conservative solution reached by standard PPO training is a local optimum of the wide-distribution objective. Policies with higher expected return on that same distribution exist. They are reachable by other routes, and the standard optimizer does not find them from scratch. Second, a regime-indexed exploration floor lets a policy trained on the fixed wide distribution from the start match or beat curriculum methods on real-world zero-shot success. It also avoids the forgetting that curriculum backtracking causes.
The mechanism adapts Goal Achievement Guided Exploration (GAGE). GAGE computes a goal-achievement score $g(\pi)=\min_i J_i(\pi)/J_i^*$, where $J_i$ is the policy's return on task-reward component $i$ and $J_i^*$ an estimate of its optimum. It then forces the policy's action standard deviation to stay above a floor proportional to $1-g(\pi)$. GAGE was tested only in IsaacLab, without randomization or hardware, and it needs a known $J^*$. We re-index $i$ over $K$ anchor dynamics regimes, which are cluster centers of the randomization distribution. $J_i(\pi)$ becomes the policy's success return in regime $i$, and $J_i^*$ becomes the best return any member of a small policy population has ever reached in regime $i$. Three things follow from this change. The floor stays high while any physics regime is being sacrificed. The estimate of $J^*$ comes from data rather than prior knowledge, which addresses GAGE's stated limitation. And the environment distribution never shrinks, so there is no backtracking for the policy to forget across.
Research Program
The program has three parts.
Diagnosis. We train PPO on the fixed wide PandaPush distribution from scratch and record its wide-distribution return. We then run two "trap tests" on that same objective.
The first is a warm-start test. We initialize from the final policy of DORAEMON, a method that gradually widens randomization subject to a success constraint, and continue training on the fixed wide distribution. If that policy keeps a higher wide-distribution return than the from-scratch policy, a better stable solution exists and the from-scratch failure is entrapment.
The second is an interpolation test. We evaluate policies along a straight line in weight space between the two solutions. A dip in return between them indicates separate basins.
If warm-started training drifts back down to the timid solution, the timid solution is the true optimum of the wide objective. In that case the problem is the objective, not the optimizer, and we would report that and stop at diagnosis.
Method. We implement the regime-indexed floor on top of PPO with a population of four seeds that share the running per-regime maxima $J_i^*$. Each member keeps its own action-noise floor. The anchor regimes are $K=16$ k-means centers over samples of the 17-dimensional parameter space. Each training episode is assigned to its nearest anchor for bookkeeping, but sampling stays uniform over the wide distribution.
The deployed policy uses the mean action. The floor therefore affects training exploration only and cannot cause action chatter on the motors. This answers the hardware concern about exploration noise raised by GAGE's simulation-only evaluation.
Analysis. We track per-regime success over training for every method. The aim is to see which regimes each method gives up, and whether DORAEMON's backtracking erases skill in extreme regimes, which its authors report as a limitation.
Baselines
The naive reference is fixed wide domain randomization with standard PPO and the default entropy bonus. DORAEMON reports this at 0% real success on PandaPush, and we rerun it with our seeds and compute budget.
The strongest existing method is DORAEMON, which maximizes the entropy of the randomization distribution subject to a minimum success rate and backtracks when success falls. It reports 60% real zero-shot success with 2.68 cm mean final distance on PandaPush (Domain Randomization via Entropy Maximization). It is the strongest competitor because it attacks the same failure from the environment side. We run it from the authors' description with the same total environment steps as our whole four-member population, and with an equal hyperparameter-search budget across methods. Beating it means higher real success over the same number of trials, together with equal or better worst-regime success in simulation.
We also include LSDR (46.7% real) and AutoDR (26.7% real), with figures as reported in the DORAEMON paper. Active Domain Randomization is a further curriculum-side comparison. It uses a particle ensemble to keep the environment sampler diverse while the policy itself is trained with standard RL.
Three ablations isolate the key ingredient. The first is a single global floor indexed by average success instead of the minimum over regimes, which is GAGE as published adapted to success. The second is a constant hand-tuned noise floor. The third is the four-seed population with shared maxima but no floor, which checks that population size alone is not the cause. Each ablation gets the same compute and tuning budget as the full method.
A specialist-distillation baseline is described under the alternatives below. No reported result was found on Lacuna for distillation on PandaPush.
Why not just…?
Why not just use a curriculum like DORAEMON, which already works? It is the main threat, and it is the head-to-head comparison. Its own authors report that backtracking causes forgetting of hard regimes. A floor that never shrinks the distribution should not suffer from this. The per-regime success curves will show whether DORAEMON actually loses those regimes and whether our method keeps them. If DORAEMON matches us on both real success and worst-regime success, the method claim fails. The diagnostic claim would still stand.
Why not just raise the entropy bonus or fix a minimum action noise? GAGE shows that standard PPO in IsaacLab humanoid running settles below 4 m/s, while goal-guided exploration reaches 7 m/s. That evidence suggests the timing of noise reduction matters, not just its level. That evidence comes from locomotion without randomization, though. The constant-floor ablation tests directly whether tying the floor to the worst regime adds anything.
Why not just train specialists per regime and distill them into one policy? This is a serious alternative. It sidesteps exploration entirely, and it would also confirm the entrapment hypothesis if it worked. We run it with the same 16 anchors, using specialists trained for an equal share of the step budget and behavior-cloned into a history-conditioned student. The concern is that regime-specific teachers act on dynamics the student cannot observe, so the student imitates an average of their actions. If distillation wins, we will report that our floor is the weaker way to escape the trap.
Why not just use a bigger network? SimBa (SimBa: Simplicity Bias for Scaling Up Parameters in Deep RL) shows that a residual architecture biased toward simple functions lets RL scale in simulation. If the collapse is a capacity problem, a SimBa policy under fixed wide randomization should fix it without any exploration change. We include a SimBa-backbone arm, with and without the floor, to separate capacity from optimization.
Proposed Experiments
The main benchmark is PandaPush in simulation with all 17 randomized parameters, plus physical transfer to a Franka Emika Panda. Success means the box ends within 3 cm of the goal, and we also report final distance. We use at least 30 real trials per method, positions fixed across methods. In simulation we report wide-distribution return and worst-of-16-regime success, each over 10 seeds with interquartile means.
As a second, lower-cost check we use ManiSkill2 PickCube (ManiSkill2), with dynamics randomization added. The ManiSkill2 paper reports 91% success in simulation and 60% real success without fine-tuning, but that figure is for its own setup, so we rerun all methods there.
The decisive checks come in order. The warm-start trap test must show a stable higher-return solution on the fixed wide objective, or the premise fails. The regime-indexed floor must then beat both the global-floor and constant-floor ablations on worst-regime success, or the regime indexing contributes nothing. Finally, real PandaPush success must reach at least DORAEMON's reported 60% under our rerun conditions, without any curriculum.
References
- Domain Randomization via Entropy Maximization
- Active Domain Randomization
- Goal Achievement Guided Exploration: Mitigating Premature Convergence in Learning Robot Control
- SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills