A robot trained with reinforcement learning usually ends up with one way of solving its task. When a motor weakens or the floor gets slippery, that one way can fail completely. A line of work on "behavioral redundancy" addresses this by training several different policies that each solve the task almost as well as the best one. At test time the agent tries them and keeps whichever survives. These repertoires are discrete: four, five, or ten policies. The perturbations they are meant to absorb are continuous, such as an obstacle at any position, friction anywhere between ice and asphalt, or partial motor loss. We propose to train a single policy conditioned on a continuous low-dimensional code $z$, so that every code in a bounded region, and therefore every straight line between two codes, is a near-optimal solution to the task. Across that region, the codes should produce behaviors that are as distinguishable as possible. Test-time adaptation then becomes a smooth, low-dimensional search over $z$ rather than a trial-and-error pick among a handful of options.
The technical crux is how the near-optimality constraint is enforced. Existing methods apply it per discrete skill (One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL, SMERL) or per policy in a set (Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality). With a continuum of codes, neither transfers directly. A single averaged constraint lets diversity pressure carve failing pockets into the code space while the average still passes. Per-episode reward gating becomes noisy and binary. Our core ingredient is a multiplier field: a small network $\lambda_\psi(z)\ge 0$ that learns, for each code, how strongly the task constraint must be enforced there. It is paired with a critic split by reward component, so the task value of each code is estimated cleanly. Because we choose a convex support for $z$, enforcing the constraint pointwise on that support is exactly what makes interpolation safe.
Core Claim
We claim three things. First, pointwise enforcement of near-optimality over a convex latent support produces a policy family in which the fraction of codes, and of interpolation paths, achieving at least an $\alpha$-fraction of optimal return is substantially higher than under averaged constraints or reward gating. Second, it achieves this at comparable diversity. Third, under continuously parameterized test-time perturbations, the best code moves smoothly with the perturbation. A continuous family searched with a few-episode budget should therefore beat a discrete repertoire of matched size.
Written out, the objective is
$$\max_{\pi}\; I(S;Z)\quad \text{s.t.}\quad J(\pi_z)\ge \alpha\, J^*\;\;\text{for all } z\in[-1,1]^d .$$
Here $\pi_z$ is the policy $\pi(a\mid s,z)$ with the code held fixed, $J$ is expected extrinsic return, $J^*$ is the return of a separately trained single-policy agent, $\alpha\in(0,1)$ is the tolerance, and $d$ is between 2 and 4. We relax it to the Lagrangian
$$\mathcal{L}(\pi,\psi)=\mathbb{E}_{z\sim U([-1,1]^d)}\Big[\hat I_z(\pi)+\lambda_\psi(z)\big(J(\pi_z)-\alpha J^*\big)\Big].$$
In this expression, $\hat I_z$ is the per-code contribution of a variational mutual-information estimate. The policy ascends $\mathcal{L}$ and the multiplier network descends it, so $\lambda_\psi$ grows only where codes violate the constraint. The uniform hypercube prior is deliberate. Any segment between two codes lies inside the support, so a constraint satisfied everywhere on the support covers every linear interpolant. This removes the "optimality valley" that non-convex policy landscapes otherwise create between near-optimal endpoints.
Research Program
The method reuses standard machinery and changes one thing. The policy and diversity estimator come from mutual-information skill discovery: an agent rewarded for visiting states from which its code can be recovered. The classic version is Diversity is All You Need: Learning Skills without a Reward Function, which uses a classifier $q_\phi(z\mid s)$ and discrete codes. For continuous $z$ we use a contrastive estimator over state transitions, in the style of CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery and Behavior Contrastive Learning for Unsupervised Skill Discovery. Those papers show that contrastive estimators avoid the "static posing" failure, where each skill simply holds a distinct pose.
Both SMERL and the bottleneck analysis in Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills warn that unconstrained diversity drifts toward task-irrelevant or unreachable behavior. Here the task constraint plays the role of an anchor. EDL's continuous interpolation was never tied to task return, and our method is designed to supply exactly that tie.
The critic follows the lesson of Multi-Critic Actor Learning: Teaching RL Policies to Act with Style. That paper found that a single shared critic averages conflicting value signals across styles and destabilizes the actor, and it fixed this with one critic per style. One critic per style cannot scale to a continuum. In our setting, however, the conflict is between reward components (task versus diversity), not between codes. We therefore train two critics conditioned on $z$: one for task return and one for the intrinsic reward. The task critic evaluated at a code directly supplies the per-code estimate of $J(\pi_z)$ that the multiplier field needs.
We will build this on top of soft actor-critic, the off-policy actor-critic used by DIAYN and SMERL. We will then analyze three things. The first is the coverage map of return over the code space. The second is how the multiplier field concentrates on boundary regions during training. The third is whether the optimal code under a perturbation varies smoothly with the perturbation's magnitude.
Baselines
The naive reference points are a single soft actor-critic policy trained only on task reward, and the same policy fine-tuned online after the perturbation. The single policy anchors DOMiNO's reported comparison. For fine-tuning, Adaptable Agent Populations via a Generative Model of Policies reports that gradient fine-tuning reached a lifetime score of 55.2 after 2,000 episodes in Farmworld, while latent search reached 158 in 200 episodes. We expect a similar gap but will measure it on our own tasks.
The strongest discrete competitors are SMERL and DOMiNO. SMERL uses mutual-information diversity with a per-skill return threshold. DOMiNO poses diversity as a constrained problem over state-action occupancy with an $\alpha$-optimality constraint, and reports 1.5x to 2.0x higher reward than a single-policy baseline as perturbations become severe on DeepMind Control walker tasks. That is the number to beat. For SMERL, no reported numeric result was found on Lacuna; its results are relative-performance plots. We will run both from their published descriptions at repertoire sizes $k\in\{5,10,50\}$, with the same environment-step budget, the same $\alpha$, and an equal hyperparameter-search budget.
The strongest continuous competitor is ADAP. It is a generator conditioned on a code on a sphere, trained with a pairwise action-KL diversity term and adapted by derivative-free latent search. We run it with the same latent dimension and search procedure as ours.
The ablations isolate the key ingredient by keeping the continuous prior, the contrastive estimator, and the critics fixed and changing only the constraint mechanism. The variants are: no constraint; a single global multiplier (an averaged constraint); SMERL-style per-episode gating applied to sampled continuous codes; linear reward scalarization over a sweep of weights; and a single combined critic instead of split critics. Beating the baselines means two things. The first is a higher near-optimal coverage fraction at equal or higher diversity than every ablation. The second is a higher mean return than the best discrete repertoire under a matched few-shot trial budget. Given DOMiNO's reported number, we should match or exceed a 1.5x to 2.0x gain over the single policy at the severe end of its perturbation range.
Why not just…?
Why not just run SMERL with a continuous code? This is the most obvious version of the idea and the real threat. Per-episode gating on a continuum is noisy: each sampled code gets roughly one rollout, and the binary gate cannot say how far a region is from feasibility. We expect this to leave failing pockets, but that is a hypothesis, not a finding. The gating ablation settles it directly. If gating matches the multiplier field on coverage and adaptation, our contribution reduces to "use a continuous prior," and we will report it that way.
Why not just add task reward and a weighted diversity bonus? DOMiNO reports that linear scalarization fails to trace the reward-diversity Pareto front. In its experiments, small weights collapse to one mode and large weights abandon the task. Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policies likewise reports delicate tuning of this weight. We include the sweep anyway, so the claim is checked on our tasks rather than assumed.
Why not just train a larger discrete repertoire and pick the best? With enough policies, a discrete set approximates a continuum. The cost appears at test time: under a budget of $K$ trial episodes, $k>K$ policies cannot all be tried. A continuous family lets a local search exploit smoothness. The matched-budget comparison against DOMiNO and SMERL at $k=50$ tests this. If a large discrete set wins at small $K$, the advantage of continuity is illusory.
Why not just use ADAP's continuous latent with domain randomization? ADAP's KL term pushes actions apart without requiring intermediate codes to stay functional, and the paper notes non-functional "bad apple" policies. Domain randomization requires anticipating the perturbation family; SMERL argues it fails on unmodeled shifts. We run ADAP directly, and we report domain randomization over training perturbations evaluated on held-out ones. This makes clear which regime each approach wins in.
Proposed Experiments
The primary benchmark is DeepMind Control walker.stand and walker.walk (DeepMind Control Suite), using DOMiNO's perturbations of joint damping, friction, and limb length. Each perturbation is swept on a fine grid, so both smoothness and severity can be read off.
The secondary suite is SMERL's setting: HalfCheetah, Hopper, and Walker2d under external joint forces and motor failure, and a 2D navigation task where the blocking obstacle is placed at continuously varying positions. That last task is the cleanest test of whether intermediate codes solve intermediate problems. As a stretch, we test ManiSkill Avoid. There, BMD reports preserving up to 10 distinct successful modes where standard RL kept one or two, and we ask whether a continuous family covers the space of paths between those modes.
We report four measures. Near-optimal coverage is the fraction of a dense code grid, and of random interpolation segments, achieving $J\ge\alpha J^*$. Diversity is measured by a held-out contrastive estimator and by state-occupancy spread. Few-shot adaptation is return under a budget of $K\in\{1,3,10\}$ trial episodes, with codes chosen by cross-entropy search. Smoothness is the correlation between perturbation magnitude and the location of the best code.
Three decisive checks determine whether the idea stands. The multiplier field must raise interpolation-segment coverage over the gating and global-multiplier ablations at matched diversity; if it does not, the core mechanism fails. On the obstacle-position sweep, the best code must track obstacle position smoothly, and the continuous family must beat the best discrete repertoire at $K\le 3$. Finally, on walker perturbations, our method should reach at least DOMiNO's reported 1.5x to 2.0x gain over the single policy at severe perturbations. If it falls clearly short, continuity is not buying robustness, and the proposal's main claim is false.
References
- One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
- Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality
- Diversity is All You Need: Learning Skills without a Reward Function
- CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery
- Behavior Contrastive Learning for Unsupervised Skill Discovery
- Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
- Multi-Critic Actor Learning: Teaching RL Policies to Act with Style
- Adaptable Agent Populations via a Generative Model of Policies
- Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policies
- DeepMind Control Suite