Population Disagreement as an Active Curriculum for Evolved Policy Initialization in Domain Randomization

Robots trained in simulation often fail on real hardware because the simulator gets friction, mass, or motor delay slightly wrong. The standard remedy is domain randomization: train across many randomly perturbed simulators so the policy cannot rely on any one simulator's quirks. Randomizing uniformly wastes most of the simulation budget on easy settings and tends to produce stiff, conservative behavior. "Active" variants try to spend simulation on the settings that matter. They need a signal for what matters, and that signal is weakest exactly when it is needed most, at the start of training, when the policy is still random. This note proposes a single mechanism that attacks both problems. We run a small evolutionary search over a low-dimensional part of the policy's starting parameters. The disagreement among members of that evolving population, meaning how differently they perform in a given simulator setting, then serves as the active-learning signal for which simulator settings to roll out next.

The two-sentence version is this. An evolving population of candidate policy initializations is itself a committee, and simulator settings where the committee members disagree are the ones that best tell good starting points from bad ones. We use that disagreement to choose which randomized environments to simulate during evolution. We then hand both the winning initialization and the learned "where policies differ" map to an ordinary gradient-based reinforcement learner.

Core Claim

Active domain randomization methods need a difficulty signal that is unreliable early in training. Active Domain Randomization trains a separate sampler policy (via Stein variational policy gradient) rewarded by a discriminator that compares trajectories against a fixed nominal "reference" simulator. With a random policy, the reference rollouts are themselves noise, and choosing a meaningful reference for multi-body locomotion is not obvious. PDTS instead predicts task risk with a surrogate model and adds a diversity penalty to avoid collapsing onto a tiny cluster of hard tasks, the "concentration issue". Its surrogate is also least accurate at the start.

Our claim is that disagreement across a population supplies a difficulty-and-informativeness signal from generation zero. It needs no reference simulator and no pretrained risk model. Evolution in turn benefits from active selection, because its best-known weakness is the cost and noise of evaluating fitness in stochastic simulators. We predict two effects at matched total simulator steps. First, the resulting pipeline should improve tail-risk performance, measured as the conditional value-at-risk (CVaR, the average return over the worst 10% of settings), and out-of-distribution return over PDTS. Second, it should cut run-to-run variance relative to random initialization.

The evolved object is deliberately small. Directly encoding millions of network weights in a chromosome scales poorly, and pure neuroevolution of full policies is expensive: LyEvO needed over 31 hours on up to 1,000 CPU cores for a quadrotor and reported actuation jitter. These results rule out evolving the whole policy. Instead we follow the hybrid policy form of Sim-to-Real: Learning Agile Locomotion For Quadruped Robots, which writes the action as $a(t,o)=\bar a(t)+\pi(o)$. Here $\bar a(t)$ is an open-loop trajectory that depends only on time, and $\pi(o)$ is a learned feedback term that depends on the observation $o$. That paper hand-designed $\bar a$. We evolve it, together with per-layer initialization scales for $\pi$, which are distribution-level parameters in the spirit of Evolving Deep Convolutional Neural Networks for Hyperspectral Image Denoising. The evolved vector has tens of dimensions, a range where the covariance matrix adaptation evolution strategy (CMA-ES) works well. For tasks without periodic structure, such as LunarLander, the evolved subspace is the output-layer bias and a random low-dimensional projection of the last layer.

Research Program

The evolutionary phase maintains a CMA-ES population $\{\theta_1,\dots,\theta_N\}$ of initialization vectors. At each generation we must choose $k$ simulator parameter vectors $\xi$ (masses, frictions, damping, latency, engine strength) on which to roll out every candidate. The acquisition score for a proposed $\xi$ is

$$ s(\xi)=\widehat{\mathrm{Var}}_{i}\big[R(\theta_i,\xi)\big]-\lambda\,\min_{\xi'\in B}\,\|\xi-\xi'\|^{-1}, $$

where $R(\theta_i,\xi)$ is the return of candidate $i$ in setting $\xi$. The variance term is predicted by a small probabilistic surrogate fitted to all past rollouts, which is possible because consecutive CMA-ES generations are strongly correlated. We draw one posterior sample from that surrogate, as in PDTS, rather than tuning a confidence bonus. $B$ is the batch of settings already selected in this generation, and $\lambda$ trades informativeness against coverage. The second term plays the role of PDTS's diversity penalty and of the furthest-from-centroid rule in O-MedAL. As in PDTS, we score a large pseudo-batch of random $\xi$ and keep the top $k$ under the penalty. Generation zero uses a space-filling design, since the surrogate has no data yet.

The logic of the variance term is simple. A setting where every candidate fails, or every candidate succeeds, says nothing about which candidate to keep. Settings where candidates split are the ones that sharpen the ranking. This is query-by-committee applied to fitness evaluation.

Fitness itself is the CVaR of return over the selected settings. Because active selection biases the evaluation set toward contested settings, we also compute an importance-weighted estimate against the uniform prior and report whether the two rankings agree. After a fixed evolution budget, we take the best $\theta$ and train the feedback network $\pi$ with proximal policy optimization (PPO). During PPO, environment settings are drawn by the same acquisition rule. The disagreement is now computed across the final evolutionary population together with checkpoints of the PPO learner, which gives the curriculum a warm start instead of a burn-in period.

The analysis has three parts. We measure ranking fidelity, the Kendall correlation between the actively estimated fitness ranking and a ranking from exhaustive uniform evaluation, as a function of simulator steps. We measure whether the evolved open-loop component avoids the conservative compromise that uniform randomization is known to produce. Finally, we measure run-to-run variance across seeds, which is the instability the evolutionary-initialization literature reports for random weights (Forecasting short-term electric load using extreme learning machine with improved tree seed algorithm based on Lévy flight).

Baselines

The naive reference is uniform domain randomization with standard random initialization and PPO, where the open-loop term is either absent or set to zero. This is the setting in which ADR reported collapse at low engine strengths on LunarLander and high-variance forgetting on 8-dimensional ErgoReacher. Beating it is necessary but not interesting.

The strongest existing method is PDTS, which combines posterior sampling from a risk surrogate with a diversity penalty over large pseudo-batches. It reports more than 15% improvement on CVaR$_{0.9}$ across most MuJoCo meta-RL benchmarks, up to 73% better out-of-distribution return on LunarLander relative to standard empirical risk minimization, and up to 2.4x faster convergence on Pusher (PDTS). The file gives only relative figures. We found no reported absolute CVaR or return values on Lacuna for these environments, so we will re-run PDTS ourselves. It is the strongest competitor because it already addresses sampler collapse, and it differs from us only in where the early signal comes from and in using random initialization.

ADR is the second competitor, run with its discriminator and nominal reference setting as published. ADR reports qualitative results only: near-expert generalization across engine strengths and a tight outcome spread on ErgoPusher. We found no numeric result on Lacuna for ADR's absolute returns.

For fairness, every method receives the same total simulator-step budget, with our evolutionary rollouts charged in full against that budget. All methods use the same PPO implementation and network. Hyperparameter tuning uses the same number of trials per method on the same held-out settings, over the same randomization ranges.

Three ablations isolate the key ingredients. The first keeps evolution but evaluates fitness on uniformly drawn settings, which removes the active component. The second keeps the disagreement curriculum but replaces the evolved initialization with the best of $N$ random seeds, which removes directed global search. The third uses PDTS's own risk surrogate in place of population disagreement, which isolates the signal itself.

Beating PDTS means higher CVaR$_{0.9}$ and higher out-of-distribution return at matched simulator steps, with non-overlapping seed confidence intervals over at least ten seeds, plus lower across-seed variance.

Why not just…?

Why not just warm-start PPO on the nominal simulator and then run ADR or PDTS? This is the most serious threat. It fixes the "random policy corrupts the early signal" problem cheaply and needs no evolution. Its likely weakness is that pretraining on one nominal setting pulls the policy toward a basin fitted to that setting. Evolution instead selects starting points by their tail performance across settings. We include nominal-pretrain-then-PDTS as a baseline, with its pretraining steps charged to its budget. If it matches us on CVaR and out-of-distribution return, the evolutionary phase is not justified, and we would report the disagreement curriculum alone as the contribution.

Why not just use disagreement among an ensemble of independently trained PPO policies? This is also a real threat. An ensemble of PPO seeds is a committee too. The differences are cost and diversity: evolved members are cheap, low-dimensional perturbations spread deliberately by CMA-ES, whereas PPO seeds are expensive and may converge to similar behavior. We run exactly this variant at matched steps. If it is equally good, the lesson is that committee disagreement, not evolution, drives the gain.

Why not just evolve the whole policy, as LyEvO does? LyEvO's own reported cost (31+ hours on up to 1,000 CPU cores versus about 10 hours for deep RL on one machine) and its hardware jitter argue against this. EvoGrad also found hybrid evolution-plus-gradient methods solving 94.9% of COCO functions against 61.2% for plain CMA-ES. We include a small-scale full-policy ES run only as a compute reference.

Why not identify the simulator from real data, as EASI does? EASI evolves simulator parameters to match real transitions and reduced real Cartpole pole-angle error by about 37% relative to uniform randomization. It requires real trajectories, however, and our setting is strictly zero-shot, with no real data before deployment. The two approaches are complementary rather than competing. We do not claim to beat it where real data is available.

Proposed Experiments

We evaluate on LunarLander-v2 with randomized engine strength, Pusher-3DOF-v0, and 8-dimensional ErgoReacher, following ADR's setups, and on the MuJoCo meta-RL CVaR$_{0.9}$ suite from PDTS. For locomotion, where the open-loop term matters most, we use LocoMuJoCo environments from robfiras/ls-iq and an Isaac Gym Unitree Go2 task with randomized friction, mass, and a 15–19 ms latency range modeled after the Minitaur work. If hardware is available, a zero-shot Go2 deployment compares velocity tracking against uniform randomization and PDTS policies with no real-world tuning.

Four checks decide the outcome. First, at matched simulator steps, the full method must beat PDTS on CVaR$_{0.9}$ and on out-of-distribution LunarLander return. Second, active fitness evaluation must reach a given ranking fidelity with fewer rollouts than uniform evaluation. If it does not, the active-learning half of the claim fails. Third, the evolved-initialization variants must show lower across-seed variance than both random-seed and nominal-pretrain variants. Fourth, the two ensemble-disagreement ablations must fall short of the evolved population. If they do not, the contribution narrows to the curriculum signal.

On locomotion, we additionally check whether the evolved $\bar a(t)$ yields faster, less conservative gaits than PPO learning the full action. That comparison directly tests whether global search over the open-loop structure escapes the worst-case compromise attributed to uniform randomization.