When a legged robot loses a motor or meets a slippery floor, the gait it was trained for may stop working. A good remedy is to keep a repertoire of different gaits that all solve the nominal task well, then switch to whichever one still works. Current repertoire methods learn a handful of separate policies, for example eight or ten, and pick among them by trying each one on the damaged robot. Both choices are crude. The perturbations are continuous, so the best response often lies between two stored gaits. Trying the gaits one by one is also slow and risky on broken hardware.
We propose to describe each behavior by its successor-feature fingerprint: the discounted average of the states the policy will visit, summarized as one vector. We then train a single policy that takes a target fingerprint as input and produces the behavior with that long-run footprint. Because the input space measures where behaviors lead, rather than how their joint angles look in the short term, straight-line paths between two good gaits should pass through intermediate gaits that are also good. Search over this space after damage should also behave like a smooth, low-dimensional optimization problem. Recovery then becomes a few rounds of Bayesian optimization over a continuous fingerprint, not K blind trials of discrete policies.
Core Claim
The claim has two parts, and each can be falsified. First, if the conditioning space of a skill-conditioned policy is the policy's own successor-feature fingerprint, and the policy is trained under a near-optimality constraint at interpolated points as well as at stored modes, then interpolated behaviors stay near-optimal far more often than in a latent space trained by trajectory reconstruction. Second, this geometry makes post-damage return a smooth function of the conditioning vector. A Gaussian-process search over fingerprints should therefore recover more return, in fewer trial episodes, than K-shot selection from a discrete diverse library. It should also beat a continuous skill space indexed by hand-engineered descriptors.
Some terms need defining. A successor feature $\psi^\pi(s)$ is the expected discounted sum of state features $\phi(s_t)$ along future steps of policy $\pi$ started from $s$. We define the fingerprint as $\Psi_\pi = \frac{1}{|\mathcal{C}|}\sum_{s\in\mathcal{C}} \psi^\pi(s)$, averaged over a fixed set $\mathcal{C}$ of canonical start states, exactly the descriptor that π2vec uses. π2vec only uses the fingerprint passively, to rank existing black-box policies offline; its positive real-robot rank correlation, where action-based comparison was negative, is our evidence that this vector tracks what matters behaviorally. We turn the same vector into an address that a policy can be asked to realize.
Research Program
The work has three stages: discover, densify, and adapt.
Discover. We first obtain a small diverse library of near-optimal policies with a constrained diversity method. DSP adds policies one at a time, each maximizing successor-feature dissimilarity to its predecessors subject to return above a fraction $\alpha$ of optimal. DOMiNO does the same jointly over occupancy measures, using a Van der Waals force that combines short-range repulsion with long-range attraction. We reuse these as the method for this stage, with $\phi$ set to the raw proprioceptive observation (or a fixed random projection of it) rather than engineered descriptors. Both papers show that this kind of $\phi$ yields qualitatively distinct gaits, such as eight standing behaviors in walker.stand and three-legged stances in dog. The library's fingerprints $\{\Psi_{\pi_k}\}$ give the anchor points of the space.
Densify. Next we train one conditioned policy $\pi(a\mid s,z)$ and a conditioned successor-feature critic $\psi(s,a,z)$, with $z$ sampled from the convex hull of anchor fingerprints, so that most training targets are interpolants rather than anchors. The objective asks the policy to realize its address while staying good:
$$\min_{\pi}\; \mathbb{E}_{z}\,\big\|\Psi_{\pi_z}-z\big\|^2 \quad \text{s.t.}\quad J(\pi_z)\ \ge\ \alpha J^* \ \ \text{for every sampled } z,$$
Here $\Psi_{\pi_z}$ is the realized fingerprint estimated by the critic, $J$ is task return, and $J^*$ is the best library return. We enforce the constraint with a per-$z$ Lagrange multiplier network, the device QDAC uses with hand-crafted descriptor targets. Anchors are also behavior-cloned from library policies to stabilize training.
Two failure modes are expected, and we will instrument both. Some interpolated fingerprints may be physically unrealizable at near-optimal return. The multiplier will flag these, and we report the feasible fraction of the hull rather than hiding it. Learned successor features can also collapse or rescale over training, as SFK found when it moved to categorical approximators. Holding $\phi$ fixed largely avoids this, and we keep categorical successor features as a fallback.
Adapt. At test time under an unknown perturbation, we treat return as an unknown function of $z$ and run Gaussian-process Bayesian optimization over the feasible hull. The prior mean is nominal return and the kernel uses Euclidean distance in fingerprint space. Each trial is one short episode. This echoes Fingerprint Policy Optimisation's use of Gaussian processes over state-visitation fingerprints, which the file describes but which we could not locate on Lacuna. The difference is that our search space is generative: any point queried is a policy we can execute, not one of a fixed set.
Baselines
The naive reference point is the single best nominal policy run unchanged under perturbation, plus a seed ensemble of independently trained policies with best-of-K selection. SMERL and DSP report that independent seeds collapse onto the same mode, so this tells us how much any repertoire buys.
The strongest discrete competitor is DOMiNO with K-shot selection. We use the same library our method densifies, so the only difference is continuity plus Gaussian-process search, and we report adaptation return against number of trial episodes from 1 to K. DOMiNO reports that picking the best policy from its set gave "1.5x to 2.0x higher rewards" than a single-policy baseline under joint damping, friction and limb-length perturbations in DMControl. Beating it means matching its best-of-K return using fewer than K trials, or exceeding it at K trials on perturbations whose optimum lies between modes.
The strongest continuous competitor is QDAC. It trains a single policy conditioned on a continuous target in a hand-engineered descriptor space (feet contact, velocity, jump height) using a successor-feature critic and automatic Lagrange multipliers. It reports 15% more diverse behaviors and 38% higher performance than PPGA and DCG-ME on Brax tasks, and qualitative recovery under Humanoid motor failure without a scalar figure. We run QDAC with its published descriptors and give it the same Gaussian-process search over its descriptor space, the same environment-step budget, and the same hyperparameter sweep size. Beating it means equal or better recovery with no hand-designed descriptors. A tie with hand-engineered descriptors would also be a meaningful result, since it shows the descriptors can be replaced.
Two ablations isolate the key ingredient. First, we replace the fingerprint address with a latent code from a trajectory autoencoder trained on library rollouts, in the style of SeCTAR and LAT, keeping the same constraint and search. Second, we keep the fingerprint address but drop the constraint at interpolants, enforcing it only at anchors. We found no reported result on Lacuna for continuous-sweep adaptation regret on these benchmarks, so all continuous-sweep numbers will be produced here under one protocol.
Why not just…?
Why not just keep the discrete library and try every member? This is DOMiNO's protocol, and it works: the 1.5–2x gain is real. It has two costs we can measure. It needs K trial episodes on a possibly damaged system, and it cannot represent responses between stored modes. If the anytime curves show DOMiNO reaching our final return at K trials, and no perturbation in the sweep has an optimum between modes, continuity buys nothing. We would then report that and reframe the contribution as faster selection only.
Why not just use QDAC, which already has a continuous near-optimal skill space? This is the most serious threat, and we treat it as such. QDAC's space is continuous and constrained, but it is indexed by descriptors an engineer chose for each task, and its targets are short-horizon properties such as contact rates. Our hypothesis is that, under a fixed descriptor choice, damage modes the engineer did not anticipate leave no good direction to search, while a fingerprint over the full state does. The head-to-head on Humanoid motor failure and Ant-Wall, with identical search, settles this. If QDAC matches us on every perturbation, the honest conclusion is that learned fingerprints are a descriptor-free substitute rather than a stronger one.
Why not just interpolate in an autoencoder latent space, or blend actions directly? Riemannian geodesic skills show that linear latent paths drift off the data, and the motion-interpolation study shows that raw joint blending leaves the valid motion set. Both concern kinematic validity, not task return. Two motions can be equally plausible kinematically while one tips the robot over thirty steps later. The autoencoder ablation measures exactly this, as the fraction of midpoints satisfying $J\ge\alpha J^*$. If reconstruction latents score as well as fingerprints, our geometric argument is wrong.
Why not just train one robust policy with domain randomization? SMERL argues that randomization must anticipate the perturbation family and becomes conservative outside it. We include a randomization-trained policy over the training perturbation ranges and test it on held-out perturbation types, motor failure in particular, where we expect it to fall behind. A randomized policy that wins on held-out damage would undercut the case for any repertoire.
Proposed Experiments
The first decisive check is interpolation validity. In DMControl walker.stand, walker.walk and dog, for every pair of anchor fingerprints we decode policies at evenly spaced points along the segment. We measure three things: the realized-fingerprint error against the target, using Monte Carlo rollouts rather than the critic; the fraction of points with return at or above $\alpha J^*$; and whether realized fingerprints move monotonically along the path. We compare against the autoencoder and anchor-only ablations. The proposal requires a clearly higher feasible fraction for the full method. If midpoints on dog are mostly infeasible, the continuous-redundancy premise fails for high-dimensional bodies and we say so.
The second check is between-mode recovery. We sweep perturbations continuously: joint damping, friction and limb length as in DOMiNO, and graded motor-strength loss on one leg rather than only full failure. For each perturbation level we compute oracle-best return within the discrete library and within the feasible hull, found by dense offline search. Then we test whether Gaussian-process search attains the hull oracle within a fixed small trial budget. The proposal predicts levels where the hull oracle beats the library oracle, which should coincide with transitions between which library gait is best.
The third check is adaptation cost on Brax Humanoid-Motor Failure, Humanoid-Hurdles and Ant-Wall against QDAC with matched search. The metric is cumulative return over the first N trial episodes, the quantity that matters on real hardware.
A supporting analysis checks the smoothness assumption directly. For each perturbation we fit the Gaussian process offline and report how well fingerprint distance predicts return difference, in the spirit of π2vec's rank correlation, compared with autoencoder-latent distance and descriptor distance. If fingerprint distance does not predict perturbed-return similarity better than the alternatives, the search advantage should vanish too, which makes this the cheapest early test of the whole idea.
References
- π2vec: Policy Representation with Successor Features
- Discovering Diverse Nearly Optimal Policies with Successor Features
- Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality
- Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics
- Combining Behaviors with the Successor Features Keyboard
- One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
- Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
- Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion
- Learning Riemannian Manifolds for Geodesic Motion Skills
- Does dimensionality reduction improve the quality of motion interpolation?