Many Roads to One Place: Reward-Free Discovery of Redundant Skills

When a robot learns skills without any task reward, the usual goal is for each skill to end up somewhere different from every other skill. Mutual-information skill discovery trains a policy conditioned on a skill code (a "latent-conditioned policy" $\pi(a\mid s,z)$, one network that changes its behavior depending on an input code $z$). It rewards the policy whenever the code can be guessed from the states the policy visits. That reward has a known side effect. When two skills share a doorway or corridor, they become harder to tell apart there, so the agent learns to avoid shared passages and carves out separate territories. The result is a repertoire with one way to reach each place. If that one way is later blocked by an obstacle, a locked joint, or a wall, the repertoire has nothing to fall back on.

We propose to split the skill code into two parts. One part, $g$, says where the skill ends up. The other part, $w$, says how it gets there. The objective rewards $g$ for being readable from the final state. It rewards $w$ for being readable from the intermediate states once $g$ is fixed. A constraint forbids $w$ from changing the destination. Skills that share $g$ are therefore trained to converge on the same place by visibly different routes. A shared bottleneck stops being a penalty, because routes are compared only against routes to the same destination. The approach produces, before any task is specified, the kind of redundant repertoire that methods such as SMERL and DOMiNO obtain only after they are given a task reward.

Core Claim

Classical skill discovery, and its contrastive descendants, maximize total divergence between skills. Can a MISL Fly? shows this objective contains an explicit anti-exploration term that penalizes overlap at bottlenecks. Its Contrastive Successor Features (CSF) method removes that term to improve coverage, but CSF still produces one direction or region per skill. The bottleneck analyses in Unsupervised Skill Discovery via Recurrent Skill Training and Explore, Discover and Learn motivate the same diagnosis. Redundancy methods take the opposite stance. SMERL maximizes skill diversity only within a margin of the optimal task return. DOMiNO maximizes distances between policies' state-action visitation distributions subject to $R(\pi)\ge\alpha R^*$. Both need a task reward and the optimal return $R^*$ to anchor "same outcome."

Our claim is that "same outcome" can be defined without reward: two skills share an outcome if they share a destination code. Redundancy can then be learned with a conditional mutual-information objective under an endpoint-invariance constraint. Concretely, with $s_T$ the final state of a skill segment, $s_{1:T}$ its intermediate states, and $\phi$ a learned state representation, we solve

$$\max_{\pi}\; I(\phi(s_T);\, g) \;+\; \beta\, I(\phi(s_{1:T});\, w \mid g) \quad \text{s.t.} \quad I(\phi(s_T);\, w \mid g) \le \epsilon .$$

Here $\beta$ weights route diversity and $\epsilon$ is the tolerated leakage of the route code into the destination.

The first term is a coverage objective of the CSF type. The second rewards distinct routes. The constraint makes those routes redundant rather than simply more skills going to new places. Both diversity terms are measured in $\phi$, the representation CSF learns, which approximates temporal distance between states. This matters because route differences that are only posture jitter or speed changes move little in $\phi$ and earn little reward. Differences that pass through different corridors, or rely on different limbs, move far in $\phi$ and earn much more.

Research Program

The method reuses three existing pieces of machinery and adds one new one.

First, from CSF it reuses the representation $\phi$, trained by contrastive (InfoNCE) estimation on state transitions. It also reuses CSF's successor-feature critic, a vector-valued critic predicting discounted future features, with an inner-product reward that maximizes the destination term.

Second, from CIC it reuses contrastive estimation with continuous skill vectors. A second contrastive critic scores intermediate-state features against $w$, using negatives sampled with the same $g$ but different $w$. This conditional sampling of negatives is what turns the objective into conditional mutual information.

Third, from the constrained formulations of DOMiNO and QDAC it reuses a learned Lagrange multiplier. A small auxiliary network tries to predict $w$ from $\phi(s_T)$ and $g$. Its predictive log-likelihood is an upper estimate of the leakage term, and the multiplier rises whenever that estimate exceeds $\epsilon$. We use a constraint rather than a fixed penalty weight because DOMiNO documents that linear scalarization of quality and diversity is brittle.

The new piece is the factorized code together with the conditional negatives. The closest conceptual precedent is Latent Conditioned Policies for Diverse Behavioral Control, which separates style from context in a latent code. However, it does not use mutual-information objectives to overcome bottleneck repulsion, and it does not discover destinations without reward.

The analysis has three strands.

The first is mechanistic. In bottleneck layouts we will check whether skills sharing $g$ actually pass through different doorways, or whether they satisfy the route term with trivial gait differences. If the trivial solution dominates, that would show $\phi$ does not penalize jitter enough.

The second concerns weight sharing. URSA reports that a shared network lets one damaged actuator corrupt much of the repertoire. We test whether actuator-level damage leaves at least one route per destination intact. We compare a fully shared network with a variant that gives $w$ its own output heads.

The third is sensitivity to $\epsilon$ and $\beta$. These are the two quantities a user must choose, and ComSD shows that hand-tuned weighting ranges are a practical cost in this area.

Baselines

The naive reference point is a single goal-reaching policy. It is pre-trained without reward to reach destinations sampled from CSF's learned representation and has no route code. At test time it is restarted with fresh random seeds to supply "alternatives." It answers whether stochasticity alone yields usable alternative routes. Beating it means a higher success rate when the default route is blocked, at an equal test-time budget of $K$ episodes.

The strongest unsupervised competitor is CSF itself, from Can a MISL Fly?. CSF is reported to match or slightly exceed METRA on state coverage and zero-shot goal reaching. No exact reported number was found on Lacuna for CSF coverage, so we will rerun it ourselves. CSF gets the same total skill dimensionality as our $g$ plus $w$, the same pre-training steps, and the same hyperparameter search budget. Beating it requires two results together. First, no loss of coverage, judged within seed variance. Second, a significant gain in success under route-blocking and damage when the test-time selection procedure is identical.

ComSD is the second unsupervised baseline. It is the strongest reported method on URLB skill-combination tasks, where it outperformed six baselines including CIC and APS in 15 of 16 tasks. It is run under the URLB two-phase protocol with its published weighting ranges.

As task-aware reference points we run SMERL and DOMiNO. Both receive the task reward and are given our pre-training steps plus the fine-tuning budget. DOMiNO reports 1.5x to 2.0x higher reward than a single-policy baseline under severe joint-damping, friction, and limb-length perturbations. No absolute returns for SMERL were found on Lacuna. Our method sees no reward before the task is specified, so matching these methods counts as success and beating them is not required.

Three ablations isolate the key ingredient. The first removes the endpoint-invariance constraint. It tests whether $w$ then simply produces more destinations, which would be CSF with a larger code. The second replaces the conditional negatives with unconditional ones. It tests whether conditioning on $g$ is necessary. The third measures route diversity in raw state space rather than in $\phi$. It tests the claim that the temporal-distance geometry suppresses trivial redundancy.

Why not just…?

Why not just run CSF with a large continuous skill space and cluster skills by destination afterward? This is the most serious threat. A high-dimensional CSF repertoire might already contain skills that reach nearby endpoints by different routes. The mechanism in Can a MISL Fly? predicts otherwise. Skills that converge are scored as less discriminable, and CSF's directional rewards push skills apart in $\phi$ throughout their trajectories, so near-duplicate destinations with divergent routes should be rare. We test this directly. For each destination cell reached by our method, we count distinct routes in the CSF repertoire after post-hoc clustering, and we run CSF post-hoc as a full baseline in the blocking experiments. If CSF-plus-clustering matches our method there, the proposal reduces to an evaluation insight and we will report it as such.

Why not just add a route-diversity bonus to the coverage reward with a fixed weight? DOMiNO calls this the scalarization fallacy: linear combinations of objectives are hypersensitive to the weight and fail to stay near both objectives at once. Without the invariance constraint, the route bonus is also maximized most cheaply by changing the destination. Our first ablation, which drops the constraint, is exactly this alternative, with the weight swept over the same budget.

Why not just use domain randomization for robustness? In RRLS, domain randomization is strong for average-case returns across pre-specified mass, friction, and damping ranges. SMERL argues it requires the designer to anticipate each perturbation. It also offers no alternative strategy when a route is physically blocked. We include domain randomization trained on the RRLS uncertainty sets as an extra baseline on the RRLS parameter perturbations, where it should be strong. We also run it on route blocking and locked joints, which lie outside its training distribution. A real loss to domain randomization on RRLS parameter shifts would limit our claim to structural failures, and we would state that limit.

Why not just learn a single-code skill discovery method with a trajectory-level discriminator? A discriminator over whole trajectories does reward route differences. However, it rewards them on equal footing with destination differences, so it inherits the same pressure toward separate territories. Our unconditional-negatives ablation approximates this alternative and tests whether the factorization, rather than trajectory-level scoring, produces redundancy.

Proposed Experiments

URLB protocol. Pre-training follows URLB: 2 million reward-free steps, then 100,000 adaptation steps on Walker, Quadruped, and Jaco from states. This first experiment checks that factorizing the code does not hurt standard coverage and adaptation. CIC reports 79% higher returns than APS here, which sets the scale of meaningful differences.

Bottleneck mazes. For the redundancy question proper, we build multi-room MuJoCo Ant layouts with two or three doorways between rooms. This is our construction, not an existing benchmark. Metrics are state coverage and, per destination cell, the number of skills that reach it through distinct doorways. At test time we block the doorway each skill used during training. Success after $K$-episode selection follows the SMERL adaptation protocol.

Damage and perturbation. We evaluate on SMERL's MuJoCo obstacle and motor-failure settings, on QDAC's Brax Humanoid-Motor Failure and Ant-Wall tasks, on the RRLS parameter sets, and, where compute allows, on simulated A1 damage scenarios in the style of URSA. URSA reports wins over baselines in 5 of 9 simulated damage scenarios.

Decisive check. The proposal is validated if three conditions hold together. Under doorway blocking and locked joints, success with $K$-shot selection is significantly higher than both CSF and CSF-plus-clustering. Coverage is no worse than CSF's. Removing the endpoint-invariance constraint eliminates the gain. If route diversity appears in raw state space but not in $\phi$, or blocked-route success does not improve, the conditional objective is learning cosmetic redundancy and the idea fails.