Growing the Policy as the Randomization Widens

Robots are often trained entirely in a physics simulator and then deployed on real hardware without any further training. The standard way to make this work is domain randomization: during training, physical quantities such as friction, mass and motor latency are drawn at random, along with visual properties such as textures and lighting, so that the real world looks like one more random draw. The method has a known weakness. When the randomization is made wide enough to cover reality, the policy often stops learning or settles on timid behavior. On ManiSkill2 Pick-and-Place with household objects, a standard policy trained under uniform randomization reaches only 0.3% success (Rapid Motor Adaptation for Robotic Manipulator Arms). In every one of these pipelines the network is a fixed, hand-sized design.

We propose to treat network size as something the training procedure decides. Training starts with a small policy and narrow randomization. Each time the randomization range is widened, the agent checks whether the new physics is crowding out what it already knew. When it is, the agent grows the network with a function-preserving addition, meaning new weights arranged so the policy's behavior is unchanged at the moment they are added. The underlying bet is that part of the over-conservatism under wide randomization comes from a small network being forced to represent physical regimes that conflict with each other, and that growing capacity when, and only when, this conflict appears is cheaper and more robust than choosing a large network up front. The bet can be falsified, and the experiments are built around the simplest competitor that could refute it.

Core Claim

Interference is measurable during training. Let $R_{\le k}(\pi)$ be the average return of policy $\pi$ on the randomization ranges seen up to stage $k$. After training for a fixed budget on the widened range $k+1$, we compute

$$\Delta_k = R_{\le k}(\pi_{\text{before}}) - R_{\le k}(\pi_{\text{after}}),$$

which is the return the policy lost on physics it had already mastered. If $\Delta_k$ exceeds a threshold $\delta$, the network grows. Otherwise the range widens again with no change to the network. Our claim is that this rule, which couples growth to interference, reaches higher success on held-out physics, including physics outside the training range, than three alternatives at an equal number of environment steps: a fixed network of the final grown size, the same curriculum with growth on a fixed schedule, and the same curriculum with no growth. We also expect it to produce a smaller deployed policy than the fixed large network needs to match it.

The idea differs from its closest neighbors in a specific way. Deep Reinforcement Learning with Plasticity Injection adds a residual branch that starts with zero effect when returns plateau. It was tested only on stationary Atari games, with injection at hand-chosen times and locations. Wield widens randomization in stages but keeps its network fixed. Sim-to-Real Robot Learning from Pixels with Progressive Nets grows a second network column, but it does so on the physical robot using hours of real data, without randomization and with hand-sized columns. None of them ties the timing or location of growth to the widening of the simulated environment.

Research Program

The growth step reuses existing methods. For how to grow, we use the zero-effect residual construction from plasticity injection. A new branch is added as a trainable copy minus a frozen copy, so the output is identical at insertion, and the previously learned weights stay trainable. For where to grow, we borrow from Efficient Forward Architecture Search (Petridish). Petridish attaches many candidate shortcut connections at once. Stop-gradient operators keep the candidates from disturbing the parent network, and an L1 penalty prunes the candidates down to the few that most reduce the remaining error. We adapt it in one way: the candidates are trained only on rollouts from the newly added randomization slice, so the selected location is the one that best absorbs the new physics. The operation-warmup lesson of Breaking the Curse of Space Explosion also applies here. New branches get a short warmup with the old policy weights held fixed before joint training resumes. This keeps randomly initialized capacity from being judged unfairly against capacity that has already been trained.

Several pieces of prior work shape the design. The search literature motivates growing rather than searching all at once. Progressive Differentiable Architecture Search shows that searching over a large graph all at once collapses toward parameter-free skip connections. It also shows that architectures chosen at shallow depth transfer poorly to deep evaluation networks. Both findings argue against training one large one-shot network over all randomization and pruning it afterward. The search literature also limits what we claim. Its methods assume a stationary, deterministic validation loss, which reinforcement learning does not provide. For this reason we avoid surrogate predictors in the style of Progressive Neural Architecture Search and controllers in the style of Efficient Architecture Search by Network Transformation. We measure $\Delta_k$ with multiple seeds and fixed evaluation sets of randomization draws. The sim-to-real literature supplies the host pipeline. For manipulation we keep the two-phase recipe of RMA²: a teacher policy reads a compressed encoding of the true physical parameters, and an adapter learns to predict that encoding from deployable sensors. We grow only the teacher, and phase two is left unchanged. This means any gain can be attributed to growth rather than to a new distillation scheme. The asymmetric actor-critic of Asymmetric Actor Critic for Image-Based Robot Learning, which gives the critic privileged state, is held fixed in the same way.

Growing with frozen branches increases the forward-pass cost at every step, a limitation the file notes for progressive networks. We therefore report the deployed parameter count and FLOPs for every method. We also test a final consolidation step that distills the grown policy into a single network of matching width. If consolidation removes the advantage, that is a finding in itself: growth would then help optimization but leave the final function no better.

Baselines

The naive reference point is a fixed small network trained under uniform randomization at the full target range from the first step. This is the setting that gives 0.3% success on YCB Pick-and-Place in RMA² and that over-conservatism reports describe.

The most dangerous baseline is a fixed network trained with the same widening curriculum and the same total environment steps, with its size set to the final size our method grows to on that task. This baseline isolates the claim that when capacity arrives matters, as opposed to how much capacity there is. To keep the comparison fair, it gets the same tuning budget, measured as an equal number of hyperparameter configurations, on the same learning-rate and entropy grids. Our method beats it if it reaches higher held-out success with overlapping-free 95% confidence intervals across at least five seeds. Matching it with a smaller deployed policy counts as a weaker positive result. We also run a second fixed large network at twice the final grown size, to rule out the possibility that the matched size happens to be unlucky.

The strongest existing method on the manipulation tasks is RMA². It reports 73.8% success on ManiSkill2 YCB Pick-and-Place, 90.5% on unseen EGAD objects and 51.6% on 3 mm Peg Insertion (Rapid Motor Adaptation for Robotic Manipulator Arms). We rerun it ourselves under our step budget rather than quoting those numbers, and our method uses its pipeline with only the teacher grown. Beating it means higher adapter success at the same environment steps. For locomotion, the strongest available reference is the uniform and adaptive randomization methods in Online vs. Offline Adaptive Domain Randomization Benchmark. No reported return values for these methods were found on Lacuna, so we run uniform randomization ourselves as the anchor.

Three ablations isolate the key ingredient. The first grows the network on a fixed schedule at the same stages. The second triggers growth on a plateau in overall return, which is the original plasticity-injection signal, instead of on $\Delta_k$. The third grows at a random location instead of the Petridish-selected one. A fourth ablation replaces function-preserving growth with resetting the last layers, which tests whether the benefit is simply restored plasticity.

Why not just…?

Why not just use a bigger fixed network with the same curriculum? This is the real threat. Plasticity injection reports matching the final returns of large networks trained from scratch while saving wall-clock time, which suggests that growth may win on compute without winning on performance. The matched-size baseline above settles the question directly. If it ties our method on held-out success, we will report growth as a compute and size saving rather than a robustness gain, and the interference trigger becomes a cheap way to choose the network size.

Why not just widen the randomization gradually and keep the small network? A curriculum alone may remove most of the over-conservatism. That would mean the problem is optimization, and capacity has little to do with it. The no-growth curriculum arm measures this. The diagnostic $\Delta_k$ tells the two explanations apart. If the small network shows near-zero interference at every stage, then capacity was never the bottleneck, and the proposal's premise fails on that task.

Why not just periodically reset or inject plasticity, as in plasticity injection? Atari results show that injection helps when the agent has lost plasticity, but not when it is failing to explore, as in Assault. The schedule-triggered and plateau-triggered ablations test whether the interference signal adds anything beyond injection on a timer.

Why not just use RMA²-style distillation and leave architectures alone? RMA² reports a persistent gap between its adapter and the privileged teacher. Its authors suggest that this gap may need better architectures, but they locate the gap in the adapter, not the teacher. Our method grows the teacher, so it may not close that gap. We will report teacher and adapter success separately. If teacher growth helps only the teacher, the honest conclusion is that growth belongs in phase two, and a follow-up study would apply it there.

Proposed Experiments

For state-based locomotion we use MuJoCo Hopper, HalfCheetah, Walker2D and Humanoid, with physical parameters randomized as in the adaptive randomization benchmark. Training ranges widen over five stages. Evaluation uses parameter values about 20% beyond the final training range, plus the benchmark's Noisy and Unmodeled scenarios. The Unmodeled scenario hides some parameters from the randomization, which serves as a proxy for unknown reality. For manipulation we use ManiSkill2 PickCube, YCB Pick-and-Place and Peg Insertion inside the RMA² pipeline, with dynamics and visual randomization widened in stages. The metrics are success rate or return on held-out physics, $\Delta_k$ at each stage, deployed parameters and FLOPs, and environment steps to reach a target success rate. If hardware is available, PickCube offers a physical check. ManiSkill2 reports 91% success in simulation and 60% zero-shot on a real robot (ManiSkill2). Our method would need to exceed 60% on hardware at matched deployed size.

Two checks decide whether the claim holds. The first is the comparison against the fixed curriculum network at matched final size, on out-of-range physics. The second asks whether $\Delta_k$ predicts where growth helps: tasks and stages with high measured interference should be the ones where growth beats the fixed network, and low-interference tasks should show no difference. If the first check fails, the contribution is limited to compute savings. If the second fails, the trigger carries no information, and growth would be no better than a schedule.