Growing the Neural Manifold: Curricula for Domain Randomization Guided by a Recurrent Policy's Internal State

Robots are often trained in a physics simulator and then deployed on real hardware without further training. To make this work, practitioners use domain randomization: during training, they vary the simulator's physics (friction, mass, motor strength, sensor delay) so the learned controller copes with whatever the real world turns out to be. The controllers that transfer best keep a memory of recent interaction. These are recurrent neural networks, whose internal state lets them infer unseen physical properties on the fly. The hard part is deciding how wide the randomization should be at each point in training. If the ranges are too wide too early, the policy stalls or forgets what it knew. If they grow too cautiously, compute is wasted. Current schedulers decide from outside the network: they watch reward, learning progress, or how different the robot's trajectories look.

We propose deciding from inside the network. Work on simulated motor cortex shows that a recurrent network confines its activity to a low-dimensional subspace, its "neural manifold". It adapts readily when a new demand can be met within that subspace. When the demand requires activity outside it, the network fails under noisy feedback unless the change is introduced gradually. Our claim is that the same holds for a robot policy under domain randomization. Widening a physics range is cheap when the policy can absorb it inside its current manifold. It is expensive, and causes forgetting, when absorbing it requires leaving the manifold. We can measure this before committing to the widening and then schedule randomization on that basis. This is a testable prediction about the policy's internals. It is also a scheduler that should beat reward-driven curricula, specifically on the failure modes those curricula are known to have.

Core Claim

The claim has two parts, and they can fail separately. The first is diagnostic. For a recurrent policy trained under randomized dynamics, we define out-of-manifold demand as the fraction of the change in hidden state that falls outside the current principal subspace of recurrent activity. We claim this quantity predicts, before any widening is applied, whether widening a given physics range will degrade performance on ranges already mastered. We further claim it predicts this better than reward change or learning progress do. The second part is interventional. A curriculum that widens ranges quickly where demand is low and slowly where it is high should reach broader robustness per environment step than uniform randomization or reward- and progress-driven schedulers. The gain should be largest when many physics parameters are randomized at once.

Where This Sits in Prior Work

Three papers motivate the proposal. Neural manifold under plasticity in a goal driven learning behaviour trains recurrent networks on simulated brain-computer-interface reaching tasks. It shows that perturbations requiring activity outside the existing manifold cannot be learned under imperfect feedback, except when the perturbation is stepped up gradually. That work has no physics engine, no robot, and no randomization. We carry its within-versus-outside distinction into policy learning and turn its "go gradually" finding into a signal that tells us where gradualness is needed. Motor Cortex Latent Dynamics Encode Spatial and Temporal Arm Movement Parameters Independently shows that motor-cortex activity and stability-optimized recurrent networks keep movement geometry in a dominant low-dimensional structure and treat speed as a rescaling along it. This supports the premise that a trained controller's activity really is low-dimensional and structured. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization shows why the recurrent setting matters. With uniform randomization, a long short-term memory (LSTM) policy reaches 0.89 ± 0.06 real-robot success on Fetch puck pushing, against 0.67 for a feedforward policy with randomization. Its randomization distribution was fixed throughout training, and it never examined the policy's internal state.

Three further papers schedule randomization and supply our main competitors. Active Domain Randomization proposes environment parameters using a particle-based policy search. It rewards the search with a discriminator that compares trajectories in randomized and reference environments. It also documents that uniform sampling becomes high-variance and forgets in an 8-parameter space. Reinforcement Learning with Adaptive Curriculum Dynamics Randomization for Fault-Tolerant Robot Control widens an actuator-failure range when reward crosses a threshold on Ant-v2. It finds, surprisingly, that starting from total failure and easing toward healthy motors ("hard-to-easy") beats the usual easy-to-hard order. TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL finds that a prior-free teacher is the most reliable choice. That teacher is ALP-GMM, which fits a Gaussian mixture over task parameters and samples where absolute learning progress is high. The same paper finds that ADR-style and adversarial teachers fall below random sampling on its "forgetting student" and "rugged difficulty" challenges. All three schedulers read only behavior: reward, progress, or trajectories. None of them looks at the policy's representation. That is the precise difference we test.

HACL: History-Aware Curriculum Learning for Fast Locomotion is the other nearby work. It uses a recurrent network as an external scheduler over velocity commands, not over physics, and the low-level policy's internals play no role. Its hardware results constrain us: simulated speed of 6.7 m/s dropped to 4.1 m/s on hardware. Our method does not fix unmodeled thermal limits, and we do not claim it does.

Research Program

The policy is a gated recurrent network trained with proximal policy optimization under randomized physics. Every $T$ updates, we collect hidden states $h$ on probe rollouts drawn from the current randomization ranges. We fit a principal subspace $U$ capturing 90% of their variance.

For each candidate widening $\xi$ (for example, extending the friction range by one step), we measure what absorbing it would require. We take a few gradient steps on a throwaway copy of the policy, using rollouts from the widened region. We then record the resulting change $\Delta h$ in hidden states on the probe rollouts. Out-of-manifold demand is

$$D(\xi) = \frac{\lVert (I - UU^\top)\,\Delta h \rVert^2}{\lVert \Delta h \rVert^2},$$

the share of the required state change that lies outside the current manifold. It ranges from 0, meaning the change is entirely within the manifold, to 1, meaning it is entirely outside. Massively parallel simulation makes these probe updates affordable. We budget them at 10% of environment steps and charge that cost against our method in every comparison.

The scheduler keeps per-parameter ranges. It widens a range by a step size that shrinks as $D(\xi)$ grows. It combines this with learning progress in one of two ways. In the first variant, manifold demand only gates widening. In the second, it reweights ALP-GMM's sampling. This way we can report whether the internal signal replaces the progress signal or adds to it.

A second analysis targets the hard-to-easy puzzle. The manifold view makes a specific prediction. Hard-to-easy should win when the subspace learned under severe conditions contains the subspace needed under mild ones, so that easing is a within-manifold change. It should lose when the containment runs the other way. We measure this by the principal angles between subspaces learned at the two ends of the actuator-failure range. We run the measurement on the Ant-v2 setup of the adaptive-curriculum paper. Then we repeat it with friction and sensor delay as the curriculum variable, to see whether hard-to-easy is special to actuator failure.

Baselines

The naive reference is an LSTM policy with uniform randomization over the full target ranges from the start. This is the recipe of Sim-to-Real Transfer of Robotic Control with Dynamics Randomization, whose reported real-robot figure is 0.89 ± 0.06 success on Fetch pushing. We will not have that robot, so we reimplement the recipe on our tasks rather than compare numbers across setups.

A second naive reference is a linear expansion schedule. It widens every range at a constant rate so that full width is reached at the same point in training as our method. This isolates whether the gain comes from being gradual at all or from choosing where to be gradual.

The strongest existing competitor is ALP-GMM over the physics parameters, run through the TeachMyAgent codebase. It is the strongest because TeachMyAgent found it the most reliable teacher when no expert knowledge is available, which is our setting. We found no reported result on Lacuna for ALP-GMM applied to dynamics randomization on Ant-v2, ErgoReacher, or a legged robot, so we establish that number ourselves.

We also run the scheduler of Active Domain Randomization and the reward-threshold scheduler of Reinforcement Learning with Adaptive Curriculum Dynamics Randomization for Fault-Tolerant Robot Control, in both of its directions. Neither paper's summary in our file gives scalar values, so again no reported number was found. Finally, we include a feedforward policy with stacked observation history under the best scheduler, to confirm that recurrence still matters in our tasks.

Fairness rules apply to all methods. Every method gets the same policy architecture, the same simulator, the same total environment steps (including our probe cost), the same target ranges, and the same hyperparameter search budget of 20 configurations. We use 10 seeds per method.

Two ablations isolate the key ingredient. In the first, $U$ is replaced by a random subspace of equal dimension. This tests whether the manifold specifically matters or whether any state-change statistic would do. In the second, $D(\xi)$ is replaced by the raw norm of $\Delta h$. This tests whether "large change" alone explains the effect.

To beat ALP-GMM, our method must do two things at matched steps, both with non-overlapping 95% confidence intervals across seeds. It must achieve higher return on a held-out test set covering the full target ranges. It must also achieve a higher worst-decile return, since worst-case robustness is what transfer needs.

Why not just…?

Why not just use ALP-GMM on the randomization parameters? This is the real threat. Learning progress may already capture most of what manifold demand does, because a widening that would cause forgetting often shows up quickly as negative progress. Our reason to expect a difference is timing. Progress is measured after the policy has trained in the new region, and by then any forgetting has already happened. $D(\xi)$ is measured on a throwaway copy before the widening is committed. The first decisive check below tests exactly this lead time. If manifold demand predicts forgetting no better than progress does, we will report that the internal signal is redundant and not push the scheduler.

Why not just widen everything slowly? The finding in Neural manifold under plasticity in a goal driven learning behaviour is that incremental change is enough. A uniformly slow schedule might therefore capture the whole benefit. The linear-expansion baseline tests this directly. We expect it to waste steps on parameters that could have been widened at once. The advantage should grow with the number of randomized parameters, so we vary that number from 2 to 8.

Why not just use an LSTM with uniform randomization, which already transfers? It does transfer on quasi-static pushing. However, Active Domain Randomization reports high variance and forgetting under uniform sampling in 8 dimensions. If our experiments show uniform randomization matching scheduled methods in 8 dimensions with a recurrent policy, that is itself a useful negative result for the curriculum literature.

Why not just a feedforward policy with observation history? The feedforward-with-history comparison in the Fetch work favored recurrence. Our manifold measurement also needs a persistent state to be meaningful. We keep this baseline to check that the ordering holds in our tasks.

Proposed Experiments

The first decisive check tests prediction, and it gates the rest. On ErgoReacher with 8 randomized parameters, on Ant-v2 with actuator failure, friction, and mass, and on a simulated Unitree Go1 in Isaac Gym, we apply random widenings during uniform-curriculum training. For each widening we record $D(\xi)$, learning progress, and reward change beforehand. Afterward we measure the drop in return on previously mastered ranges. We compare how well each pre-widening signal predicts a drop of more than 10%, using area under the ROC curve. If $D(\xi)$ fails to beat learning progress and the random-subspace control, we stop at a diagnostic report.

If the first check passes, the second is the curriculum comparison with the baselines and fairness rules above. On Ant-v2 we also report walking distance, to match the adaptive-curriculum paper's metric.

The third is the hard-to-easy analysis. We test whether subspace containment, measured before training finishes, predicts which curriculum direction wins for each of the three curriculum variables.

The last is hardware. We transfer the best scheduled policy and the uniform-randomization LSTM to a Go1. The randomization covers friction, payload, motor strength, and injected control delay. We measure velocity-tracking error and fall rate on carpet, slope, and pebble surfaces, with 20 trials per surface. As context, HACL reports 90% success on carpet and slopes and 60% on pebbles. Its task (maximal sprinting) differs from ours, so those figures are a reference point and not a target. The hardware claim we would make is narrow: fewer falls and lower tracking error than uniform randomization at the same training budget.