Executability, Not Density: Steering Hierarchical Agents Through Interpolated Skills

Many robot-learning systems compress demonstrations into a continuous "skill space": a small vector $z$ stands for a whole multi-step behavior, and a high-level policy (the manager) picks a $z$ every few steps while a low-level policy (the worker) turns it into motor commands. The selling point of a continuous space is interpolation. Points between two known skills should give new, in-between behaviors, such as a diagonal move built from "go right" and "go up". In practice, hierarchical agents rarely get this benefit. The standard way to keep the manager from choosing nonsense is to pull it toward skills that were common in the data. That rule forbids the in-between behaviors interpolation was supposed to provide. Without the rule, the manager wanders into regions the worker cannot execute.

This note proposes to replace "was this skill common in the data?" with "can the current worker actually carry it out from here?" We make that question measurable. A frozen decoder predicts the states each skill vector should produce, and a learned critic estimates how far the worker's real rollout will diverge from that prediction. The manager is penalized by this predicted divergence instead of by distance from the data prior. The worker may be fine-tuned, but only toward honoring the frozen prediction. The prediction thus acts as a fixed contract between the two levels: its meaning does not drift even as the worker adapts.

Core Claim

Data density and executability come apart in exactly the regions where latent interpolation matters. A self-supervised, reward-free executability critic lets a manager use interpolated skills that a density prior would suppress, and still avoids skills the worker cannot perform. Freezing the state predictor while adapting the worker gives low-level adaptation without the drifting high-level target that has forced current methods to choose between rigidity and instability.

The tension is documented in the file. Accelerating Reinforcement Learning with Learned Skill Priors (SPiRL) learns a state-conditioned skill prior $p(z|s)$ from offline data and regularizes the manager toward it with a KL penalty. This keeps exploration to "common sense" behaviors, and by construction it also discourages skills that the data does not contain. Skill-Critic shows the cost: on point mazes, SPiRL was "limited to right-angle movements learned from demonstrations." Skill-Critic's fix is to fine-tune the worker online, which reintroduces non-stationarity and requires a high-level warm-up period. Taken together, these two papers motivate the problem.

Two papers supply our mechanism. Self-Consistent Trajectory Autoencoder (SeCTAR) trains a state decoder that previews the trajectory for each $z$, and makes the closed-loop policy match that preview. We reuse the decoder, but treat consistency as a feasibility signal the manager can query, not only as a training loss. Direct Preference Optimization for Primitive-Enabled Hierarchical RL (DIPPER) penalizes the manager with the worker's value $V^{\pi_L}(s,g)$ so it stops proposing unreachable subgoals. DIPPER's subgoals are raw states, so "reaching the subgoal" has a natural reward. A latent skill has no such reward. Our consistency error is what gives "successfully executing $z$" a concrete meaning.

Research Program

We first pretrain, from offline demonstrations, a skill encoder, a state decoder $D(s,z)$ that predicts the next $H$ states, and an initial worker $\pi_L(a|s,z)$, as in SeCTAR and SPiRL. During downstream learning, every executed skill chunk produces a free label: the realized states $s_{1:H}$ under the current worker. We train an executability critic $C_\psi(s,z)$ to predict the resulting consistency error. Here $\psi$ are the critic's parameters, and $\hat s_{1:H} = D(s,z)$ is the decoder's prediction:

$$C_\psi(s,z) \approx \mathbb{E}_{\pi_L}\Big[\tfrac{1}{H}\textstyle\sum_{t=1}^{H} \lVert s_t - \hat s_t \rVert^2\Big].$$

The manager $\pi_H(z|s)$ maximizes its task value $Q_H(s,z)$ minus a feasibility penalty with weight $\beta$. A weak KL term to the prior with weight $\lambda$ is kept only for early exploration:

$$\max_{\pi_H}\ \mathbb{E}_{z\sim\pi_H}\big[Q_H(s,z) - \beta\, C_\psi(s,z)\big] - \lambda\, \mathrm{D}_{\mathrm{KL}}\big(\pi_H(\cdot|s)\,\|\,p(\cdot|s)\big).$$

The critic only learns about skills that were actually executed. We therefore spend a small, fixed fraction of rollouts on probe skills drawn along line segments between pairs of high-prior latents. This is the cheapest way to teach it about interpolated regions.

The worker is fine-tuned online with a dense reward equal to the negative consistency error, so it learns to realize interpolated previews it initially executes poorly. The decoder $D$ stays frozen throughout. The manager therefore always plans against fixed skill semantics. What changes as the worker improves is which skills are executable, and the critic tracks that.

The analysis has three parts. We check whether the critic's predicted error matches measured error on held-out interpolated latents. We map, over the latent space, where prior density and executability disagree. We test whether the frozen-decoder contract removes the need for Skill-Critic's warm-up period.

Baselines

The naive reference points are flat agents. Flat offline RL has already been tested on AntMaze: IQL-TD-MPC reports that CQL, AWAC, TD3-BC and Decision Transformer "all get zero or near-zero normalized evaluation scores on the medium and large antmaze tasks." We will also run flat SAC and behavior cloning on each task with the same data and environment steps, to confirm the floor in our own setup.

SPiRL (Accelerating Reinforcement Learning with Learned Skill Priors) is the most direct competitor on the density-versus-executability question, because it is exactly the density-regularized manager we claim is too conservative. Skill-Critic (Skill-Critic) is the strongest existing method for adapting the worker. It is the competitor that matters most, because it already escapes SPiRL's right-angle limitation by fine-tuning. Beating it requires showing that our gain comes from better skill selection and stability, not only from adaptation. The file reports no numerical tables for either method, so for both we will rerun the published methods.

For offline AntMaze, IQL-TD-MPC is the strongest reported hierarchical result: average normalized D4RL score "over 40" on medium and large AntMaze when its manager augments base offline learners. Our offline variant counts as beating it only if it exceeds that level averaged over the same tasks and dataset splits.

All learned methods share the same pretrained encoder, decoder and worker initialization. They also share the same skill horizon $H$, environment-step budget, network sizes, and an equal hyperparameter search budget of a fixed number of configurations per method, with five or more seeds. Beating a baseline means higher success rate or normalized score at equal steps, with non-overlapping confidence intervals on at least the interpolation-requiring tasks.

Three ablations isolate the key ingredient. The first removes the critic and keeps worker consistency fine-tuning, which is Skill-Critic-like adaptation under a frozen contract. The second keeps the critic but freezes the worker, testing selection alone. The third lets the decoder train jointly, testing whether the frozen contract is what provides stability.

Why not just…?

Why not just tune SPiRL's KL weight down? Lowering the weight on the prior lets the manager reach interpolated skills, but it also removes the only guard against unexecutable ones. The trade-off runs along a single dial, and no setting of the dial can tell a valid diagonal apart from an invalid midpoint, because the prior measures data support and nothing else. We will sweep $\lambda$ for SPiRL over the same budget we give our method and plot success against $\lambda$. If some setting of $\lambda$ matches our method, the critic is unnecessary and we will say so.

Why not just penalize decoder uncertainty, as in Riemannian geodesic methods? Learning Riemannian Manifolds for Geodesic Motion Skills weights a latent metric by a decoder's predictive variance, so high-uncertainty regions act as barriers, and produces smooth motions on a real arm. This is a serious alternative. Decoder uncertainty, however, again tracks distance from the demonstrations, not the current worker's ability, and the same paper notes that reconstructions degrade outside data support. We will implement an ensemble-variance penalty in place of $C_\psi$, using the same manager objective. If it matches the executability critic on the diagonal-maze task, our claim that density and executability diverge is wrong for these domains.

Why not just let the high-level Q-function learn feasibility from reward? In principle, an unexecutable skill earns low return and $Q_H$ learns to avoid it. Under sparse rewards, though, $Q_H$ carries almost no signal early in training. The consistency error is dense and needs no reward. SeCTAR's finding that flat RL earned no reward at all on its Object Manipulation and Swimmer tasks shows how little reward-driven signal exists in such regimes. The "critic off, worker frozen" condition, compared with our full method, measures how much faster feasibility is learned from consistency than from return alone.

Why not just use flat offline RL or behavior cloning? IQL-TD-MPC's report of near-zero flat-learner scores on medium and large AntMaze, together with D4RL's finding that behavior cloning fails on tasks requiring stitching (D4RL), makes this unlikely to compete on long-horizon tasks. We keep it only as a floor.

Proposed Experiments

The decisive test is a D4RL Point Maze variant whose demonstrations contain only axis-aligned motion. The test layouts require diagonal corridors, which is the setting where Skill-Critic observed SPiRL's right-angle failure. Here the prediction is sharp. Our method should match Skill-Critic in final success and reach it with fewer steps and without a warm-up period. It should clearly beat SPiRL at any KL weight, and its selected latents should lie measurably off the high-density modes while keeping low realized consistency error. Falsification is equally clear: if the uncertainty-penalty ablation or a tuned SPiRL matches us here, the central claim fails.

We then test transfer on Franka Kitchen, where SPiRL sequenced up to four subtasks; the file gives no reported number for Kitchen, so all methods are rerun. We also test on D4RL AntMaze medium and large. In the offline variant, the manager is trained with the critic penalty on logged chunks and compared against the reported IQL-TD-MPC average above 40. In the online variant, it is compared against our reruns of SPiRL and Skill-Critic.

Two diagnostic checks accompany every run. The first is calibration: the rank correlation between $C_\psi$ and measured consistency error on held-out probe latents, broken down by distance from the nearest data mode. The second is stability: the variance of the manager's value estimates during the first stretch of worker fine-tuning, with the frozen and unfrozen decoder compared directly. Whole-body humanoid skills, such as the hyperspherical skill space of Reference Grounded Skill Discovery, are a natural later target. We leave them out of the core program because a Euclidean squared-error consistency measure may not suit cyclic gaits.