When a robot is trained only from tidy starting poses, it learns to replay a memorized motion instead of correcting itself. Starting episodes from random, off-balance poses fixes this, because the policy has to learn to recover. Separately, methods that learn a repertoire of different near-optimal behaviors give a robot a backup plan when a motor breaks. These two ideas have never been combined in a way that preserves what makes each one useful. We propose training a single policy that takes a behavior code as an extra input, and requiring that from each random start state several distinct codes can bring the robot back to a nominal operating regime. Success is judged by a learned reachability signal rather than a hand-tuned reward formula.
The point that makes this more than "add random starts to an existing diversity method" is a specific failure we expect. When starts are random, current diversity objectives can be satisfied by assigning each behavior code its own region of start states. The repertoire then looks diverse on aggregate but offers no backup at any single start. Our objective measures diversity conditional on the start state, so a backup must exist where it is needed. We also pick training starts where backups are missing.
Core Claim
Existing redundancy methods train from fixed nominal starts. SMERL (Kumar et al.) rewards a discriminator for telling skill codes apart from visited states, but only while task return stays within $\epsilon$ of optimal. DOMiNO (Zahavy et al.) maximizes the spread of the policies' visitation distributions, subject to each reaching a fraction $\alpha$ of optimal return. BMD (Cai et al.) keeps diffusion-policy modes alive during RL fine-tuning. All of them leave open whether the modes survive off-nominal starts.
Work showing that random starts matter trains one policy with one behavior. Reda, Tao, and van de Panne report a narrow-start Walker2D policy falling from 2459.30 to 9.78 when evaluated on broad starts. Zhou et al. report a 29% drop in drift success when start randomization is removed. These results motivate the regime. The performance-difference bounds of Makkar et al. explain why a repertoire validated only at nominal starts says little about off-nominal ones: the loss from a shifted start distribution is bounded only by its distance from the training distribution.
The closest existing system is ASE (Peng et al.), which supplies part of our method. It trains a policy conditioned on a code constrained to the unit sphere and starts 10% of episodes mid-fall, so recovery is learned across the code space. ASE does not require that different codes offer alternative recoveries from the same start, and it never tests damage-time switching.
We claim three things. First, under broad starts, the state-only discriminator objective used by SMERL degrades into start-partitioning, and DOMiNO's aggregate visitation-diversity objective may as well. Second, conditioning diversity on the start state and enforcing the recovery constraint per start-state region prevents this. Third, the resulting repertoire keeps more starts recoverable after single-actuator damage than either a robust single policy or a nominally trained repertoire.
Research Program
The policy is $\pi(a \mid s, z)$, where $s$ is the state and $z$ is a code drawn from a fixed set of $K$ values. Following ASE, we also test a variant where $z$ lies on the unit sphere. Let $s_0$ be the episode's start state, drawn from a broad distribution $\rho$. Let $\mathcal{N}$ be a nominal set, such as upright standing states gathered from a nominal controller. Let $T$ be a recovery horizon and $\delta$ a tolerated failure rate. The objective is
$$\max_\pi \; I(S; Z \mid S_0) \quad \text{s.t.} \quad \Pr_\pi\big[\, s_t \in \mathcal{N} \text{ for some } t \le T \,\big|\, s_0, z \big] \ge 1-\delta \;\; \text{for all } z \text{ and all start bins } b(s_0).$$
Here $I(S;Z\mid S_0)$ is the mutual information between visited states and the code, given the start. We estimate it with a discriminator $q(z \mid s, s_0)$ that sees the start state. A code that can be identified from the start state alone therefore earns no diversity credit. The constraint is enforced with one Lagrange multiplier per start bin, as in DOMiNO's constrained formulation but localized. A code that solves only some regions is thus penalized where it fails.
The success signal replaces manual reward shaping with self-supervision, reusing the method of Mezghani et al.. That work trains an asymmetric reachability network on reward-free exploration data and builds a directed graph over states. Its dense reward combines local reachability with graph shortest-path distance. We use the same machinery to estimate steps-to-$\mathcal{N}$ and reward the decrease. The asymmetry matters here because falls are irreversible in the way their directed graph handles. The human still specifies $\mathcal{N}$, so this is a reduction, not an elimination, of hand specification. We test that cost directly against hand-shaped rewards.
The start distribution adapts during training. Bins are resampled in proportion to their redundancy deficit, meaning how far the number of codes that currently recover from that bin falls below a target $m$. This is a task-aligned answer to the open question of principled start distributions. It is deliberately simple, so its contribution can be ablated against a uniform $\rho$.
Baselines
The naive reference is a single policy trained by TD3 or PPO from nominal starts with the environment's standard hand-shaped reward. The Reda, Tao, and van de Panne Walker2D result (2459.30 on narrow starts, 9.78 on broad) tells us what to expect from it.
A stronger single-behavior baseline uses the same algorithm with broad starts and physics domain randomization, matching the Zhou et al. recipe. It isolates whether redundancy adds anything over plain robustness.
The strongest competitors are SMERL and DOMiNO retrained under exactly our broad start distribution, given the same environment steps, network size and hyperparameter search budget as our method. DOMiNO reports 1.5× to 2.0× higher reward than a single policy under severe perturbations on DeepMind Control walker tasks (Zahavy et al.). For SMERL, no reported numerical score found on Lacuna for MuJoCo. Beating these baselines means a higher per-start redundancy rate and a higher post-damage recoverable fraction at equal nominal recovery success, not merely higher aggregate diversity.
An ASE-style low-level policy, trained with mid-fall starts and code-conditioned discrimination without task conditioning, tests whether existing motor-prior training already yields alternative recoveries.
Three ablations each remove one ingredient. The first drops $s_0$ from the discriminator, which is the key ingredient. The second replaces per-bin constraints with one global constraint. The third replaces the deficit-driven start distribution with a uniform one. A fourth comparison swaps the reachability reward for the hand-shaped reward, holding everything else fixed.
Why not just…?
Why not just run DOMiNO or SMERL with randomized starts? This is the real threat, and it is the headline experiment rather than something we argue away. Our prediction is that the unconditioned discriminator scores well by tying codes to start regions. The test measures, for every held-out start, how many codes recover and how many distinct recoveries exist among them. If broad-start DOMiNO already achieves per-start redundancy comparable to ours, the conditional objective is unnecessary. In that case we would report that, and the contribution would shrink to the evaluation protocol and the reward-free constraint.
Why not just train one robust policy with domain randomization? Kumar et al. found that adversarially robust single policies become conservative and that domain randomization requires anticipating perturbations in advance. Recovery from off-nominal starts often has several valid routes, such as rolling forward or backward. A single unimodal policy averages these routes, as shown in the multimodality analyses of Zhang et al. and Huang et al.. The held-out actuator-failure test settles this, because none of those failures appear in training.
Why not just ensemble independently seeded policies? Seeds are a cheap source of variety. However, nothing pushes them apart, so they may share the same structural dependency. This is the correlated-failure problem URSA exhibited (Grillotti et al.), where large blocks of skills failed together when one limb broke. Shared conditioning is also over an order of magnitude more sample-efficient than separate networks, according to Chen et al.. We include a $K$-seed ensemble at matched total compute.
Proposed Experiments
Environments are drawn from the benchmarks in the file:
- MuJoCo HalfCheetah, Walker2d and Hopper, with SMERL's motor-disablement and joint-force tests.
- DeepMind Control
walker.standandwalker.walk, with DOMiNO's damping and friction perturbations. - PyBullet Walker2D, with the start-width parameter $k$ swept as in Reda, Tao, and van de Panne.
Brax versions of these locomotion tasks provide high throughput where needed.
The first metric is the per-start redundancy rate: the fraction of held-out starts from which at least $m$ codes reach $\mathcal{N}$ by distinct trajectories. "Distinct" means separated by a fixed occupancy distance threshold. We report the rate for $m=2$ and $m=3$.
The second metric is the post-damage recoverable fraction. For each single-actuator disablement held out of training, it is the fraction of starts from which at least one code still recovers. Code selection uses $K$-shot search, as in SMERL.
We also report nominal return and recovery time, so that robustness gains cannot hide a collapse in task quality.
The decisive checks are three. First, the start-partitioning diagnostic: how well the start state alone predicts which code is active. We expect it to be high for broad-start SMERL and DOMiNO and low for our method. Second, the gap in post-damage recoverable fraction between our method and broad-start DOMiNO as $k$ widens. Third, whether the reachability reward matches the hand-shaped reward on nominal return while giving equal or better redundancy. If the first diagnostic shows no partitioning in the baselines, the central claim is refuted.
References
- One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
- Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality
- Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policies
- Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning
- Learning to Drift With Individual Wheel Drive: Maneuvering Autonomous Vehicle at the Handling Limits
- Putting the Spotlight on the Initial State Distribution
- ASE: large-scale reusable adversarial skill embeddings for physically simulated characters
- Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping
- Latent State Marginalization as a Low-cost Approach for Improving Exploration
- Reparameterized Policy Learning for Multimodal Trajectory Optimization
- From Tabula Rasa to Emergent Abilities: Discovering Robot Skills via Reset-Free Unsupervised Quality-Diversity
- DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization