Novelty search finds many different robot behaviors by rewarding a controller for doing something unlike what has been seen before, rather than for scoring well on a task. Because it rewards difference, it also tends to reward differences that exist only because the simulator is wrong. A swarm that "mills" in a circle in simulation may simply clump together on the real floor because the simulator understated friction. A learned behavior description built from camera or range-sensor streams can treat sensor noise as if it were a new behavior. We propose to change what novelty search counts as novel. A candidate behavior earns novelty credit only when it stays recognizably the same, and recognizably different from the archive of behaviors already found, across an ensemble of deliberately perturbed simulators. The perturbations vary the physics (friction, actuator delay, mass) and the sensors (noise, dropout, range limits). The goal is a whole repertoire of distinct behaviors that transfers to hardware zero-shot, meaning with no real-world trials during the search.
The concrete change has two parts. First, the learned behavior space is trained so that two rollouts of the same controller under different perturbations count as the same behavior. Second, novelty is scored as a lower confidence bound on how far apart two behaviors are, so a behavior whose description scatters under perturbation earns little credit. If this works, it replaces two things current systems depend on: hand-tuned "under-representation" of the robot in simulation, and periodic real-robot tests to predict which behaviors will transfer.
What exists and what is missing
The closest work is Discovery and Deployment of Emergent Robot Swarm Behaviors via Representation Learning and Real2Sim2Real Transfer. It runs novelty search in a behavior space learned by contrastive learning (SimCLR, a method that trains an encoder to map augmented views of the same clip close together) on 6,000 simulated swarm videos. It closes the reality gap by hand: the authors measured the HeRo+ robots, added time-of-flight sensors, and made the simulated robots deliberately weaker than the real ones. With this tuning, 70% of discovered behaviors worked one-shot on hardware. With the default simulator, only 22% did. Behaviors such as milling and wall-following failed because of real friction the simulator did not capture. The paper motivates our proposal directly, since it shows that novelty in a learned space chases simulator artifacts. Our method differs in one specific way. The paper's encoder learns invariance to image augmentations from a single simulator. Ours learns invariance to physics and sensor perturbations, and that invariance is built into the novelty score itself.
Two other repertoire methods handle transfer differently, and both constrain the design. Evolving a Behavioral Repertoire for a Walking Robot. uses novelty search with local competition plus a regression model that predicts which controllers will transfer. That model needs about 60 real hexapod trials and is unreliable at the edges of the repertoire. From Tabula Rasa to Emergent Abilities avoids simulation entirely and learns skills on a Unitree A1 in 5 hours of real interaction. Neither involves noisy external sensors, and neither is zero-shot.
On the reality-gap side, three methods supply tools but optimize a single task policy rather than a set of diverse behaviors. Active Domain Randomization searches for diverse physics settings that make the agent's trajectories diverge. Domain Randomization via Entropy Maximization widens the randomization ranges while keeping the success rate above a threshold. SPiDR penalizes disagreement across an ensemble of perturbed simulators.
SPiDR's penalty appears to contradict novelty search, since it punishes the divergence that novelty rewards. Our answer is to separate the two kinds of divergence. Divergence between different controllers is what we want. Divergence of one controller across perturbed simulators is what we penalize. Assessing Transferability From Simulation to Reality for Reinforcement Learning names the underlying problem "simulation optimization bias", the tendency of an optimizer to exploit simulator errors. We treat novelty search as an especially aggressive case of that bias.
Core Claim
Run each candidate controller $\theta$ under $K$ sampled perturbations $\xi_1,\dots,\xi_K$, which mix physics and sensor changes. Embed each rollout with an encoder $f$ to get descriptors $z_k(\theta)$. Let $\mu_\theta$ be their mean and $\sigma_\theta$ their spread (the mean distance of the descriptors from $\mu_\theta$). The robust distance from $\theta$ to an archived behavior $j$ is
$$d(\theta, j) = \lVert \mu_\theta - \mu_j \rVert - \beta(\sigma_\theta + \sigma_j),$$
where $\beta \ge 0$ sets how pessimistic the score is. Novelty is the average of the $k$ smallest values of $d$ over the archive, which is the usual nearest-neighbor novelty score with the plain distance replaced by this bound.
The encoder $f$ is trained contrastively, with a pairing rule different from standard augmentation. Positive pairs are rollouts of the same controller under different perturbations. Negatives are rollouts of different controllers.
The claim is that these two pieces together steer search away from behaviors whose identity depends on simulator details. Such a behavior has a large spread $\sigma_\theta$, or it collapses onto existing behaviors in the invariant embedding. Either way it earns little novelty, while robust behaviors keep their full credit. Averaging over perturbations alone does not do this. A milling controller that turns into aggregation under higher friction has a mean descriptor halfway between the two, and that midpoint can look novel.
Research Program
We will build the method on top of standard novelty search, following Exploiting Open-Endedness to Solve Problems Through the Search for Novelty, with an optional quality term in the style of MAP-Elites as described in Quality Diversity: A New Frontier for Evolutionary Computation. The encoder is retrained periodically on rollouts from the current archive.
Sensor and physics perturbations are kept as separate channels so we can attribute effects to each. Sensor perturbations include range noise, dropout, and reduced range for time-of-flight or LiDAR-like sensors, and blur and noise for cameras. Physics perturbations include friction, actuator delay, and mass.
We will also measure the trade-off that domain randomization is known for. If perturbation ranges are too narrow, artifact behaviors pass the filter. If they are too wide, many behaviors collapse into one. We will sweep the range width and $\beta$ and report repertoire size against transfer rate. If time allows, we will add an extension that chooses perturbations adversarially to maximize descriptor spread, borrowing the sampler from Active Domain Randomization.
Every evaluation of whether a behavior was preserved uses ground-truth descriptors defined independently of the learned encoder. For swarms these are cluster count, angular momentum, and dispersion. For walkers, the descriptor is the final body position. This keeps the method from being graded on its own embedding.
Baselines
The naive reference point is novelty search on SimCLR embeddings in the default, untuned simulator. This is Mattson et al. without their tuning, for which they report 22% one-shot hardware success on HeRo+.
The strongest existing method is Mattson et al.'s hand-tuned under-representation pipeline, which reports 70% one-shot success and 90% with adjusted starting positions. We aim to match or beat 70% starting from the default simulator, with no hand tuning.
The most dangerous competitor is quality-diversity search under domain randomization. It uses the same $K$ perturbed rollouts per candidate, scores novelty on the mean descriptor, and uses worst-case task quality. It shares our perturbation distribution, rollout budget, and hyperparameter tuning budget. Our method beats it only if it achieves higher hardware behavior-preservation at equal repertoire size.
A second competitor adds a SPiDR-style penalty: ensemble disagreement on next-state prediction is subtracted from quality, with $\lambda$ tuned on the same budget. This tests whether penalizing dynamics disagreement is enough, without changing the behavior space.
TBR-Evolution's transferability model is run as an upper reference. It gets the 60 real trials its paper reports, which our method never sees.
Two ablations isolate the key ingredient. One uses the invariant encoder with $\beta=0$. The other keeps the bound but uses a standard augmentation-trained encoder. A further split runs the full method with sensor-only and with dynamics-only perturbations. All methods get the same total number of simulator steps, so ours evaluates $K$ times fewer candidates than plain novelty search.
Why not just…?
Why not just run quality-diversity under ordinary domain randomization? This is the real threat, and it is the main baseline above. The argument against it is specific. Averaging descriptors rewards the midpoints of unstable behaviors, and worst-case quality says nothing about whether the behavior's identity is preserved. The ablation with $\beta=0$ and the mean-descriptor baseline will show whether the bound matters. If they match the full method, we will report that domain randomization alone suffices for repertoires.
Why not just hand-tune the simulator as Mattson et al. did? It works, with 70% one-shot success. But it takes per-platform measurement, and the paper itself reports milling and wall-following failing when the tuning was off. We test whether a learned criterion reaches the same transfer rate without that engineering, and whether it filters out milling before deployment.
Why not just learn on the real robots? URSA needed 5 hours on a single quadruped with a safety critic and a supervising human. That does not scale to swarms of many robots or to platforms that break on the first failure.
Why not just use a few real trials to train a transferability predictor? TBR-Evolution needs about 60 trials and its predictor is weak at the edges of the repertoire, which is where novelty search spends its effort. We include it as a reference rather than a competitor, and measure how much of the gap the zero-shot method closes.
Proposed Experiments
The first experiment replicates the HeRo+ swarm setting, or a comparable low-cost swarm with time-of-flight sensors if HeRo+ is unavailable. We start from the default simulator and report the fraction of discovered behaviors preserved one-shot on hardware, judged by the ground-truth swarm descriptors, against the 22% and 70% reference points. We also check whether milling and wall-following are filtered out before deployment.
The second experiment uses walking robots, simulated hexapod and MuJoCo walkers. The held-out "real" targets are the physics changes from State Revisit and Re-explore: doubled gravity, friction reduced to 30%, Big Thigh, and Long Torso. To these we add held-out sensor noise and a different physics engine. We measure two things. Preserved coverage is the fraction of archive cells whose behavior lands in the same ground-truth cell in the target. Target coverage is the number of distinct cells reached in the target. No reported result was found on Lacuna for repertoire transfer on these modified MuJoCo tasks, since SR² reports only single-task returns. We will therefore run all baselines ourselves.
The decisive check has three parts. First, the full method must beat mean-descriptor domain-randomized quality-diversity on preserved coverage at matched repertoire size and compute, on both testbeds. Second, removing the invariant encoder or the bound must each cost a measurable share of that gain. Third, the sensor-only and dynamics-only runs must show which channel produced the artifact behaviors.
References
- Discovery and Deployment of Emergent Robot Swarm Behaviors via Representation Learning and Real2Sim2Real Transfer
- Evolving a Behavioral Repertoire for a Walking Robot.
- From Tabula Rasa to Emergent Abilities: Discovering Robot Skills via Reset-Free Unsupervised Quality-Diversity
- Active Domain Randomization
- Domain Randomization via Entropy Maximization
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Assessing Transferability from Simulation to Reality for Reinforcement Learning
- Exploiting Open-Endedness to Solve Problems Through the Search for Novelty
- Quality Diversity: A New Frontier for Evolutionary Computation
- State Revisit and Re-explore: Bridging Sim-to-Real Gaps in Offline-and-Online Reinforcement Learning with An Imperfect Simulator