Robots that follow instructions often organize behavior into skills: a high-level model picks a short-lived behavior ("open the drawer"), and a low-level controller carries it out. In the best-known language-conditioned systems, each skill runs for a fixed number of timesteps $H$, no matter what. It keeps running after the drawer is already open, and it gets cut off halfway down a long reach. Separately, work on multimodal embeddings has shown that language and continuous signals, when mapped into one shared vector space, end up in separated clusters. This separation is called the modality gap. It is usually treated as a property of static image–text pairs. We argue that in instruction-following control the two problems are really one. Chopping trajectories into fixed-length pieces produces trajectory embeddings that match no sentence cleanly. And a shared language–trajectory space, once its gap is corrected, already contains the signal needed to decide when a skill is finished.
The proposal is to segment demonstrations, and to end skills during execution, at the moment when the embedding of the trajectory so far stops getting closer to the language description of the current skill. We first measure how much fixed-length slicing inflates the language–trajectory gap. We then replace the fixed horizon with this alignment-driven stopping rule. The test is on the LOReL and Kitchen manipulation benchmarks, against fixed-horizon skill learners and against language-free ways of learning when to stop.
Core Claim
A trajectory segment that covers exactly one described behavior should embed closer to that behavior's language description than any shorter or longer segment. If that holds, then "how well does what I have done so far match what I was told to do" is a usable, language-grounded signal for skill termination. It needs no hand-annotated boundaries and no horizon hyperparameter. The claim has two parts, and each can fail on its own. The diagnostic part is that fixed-length slicing measurably widens the language–trajectory gap and spreads the trajectory embeddings assigned to each skill code. The method part is that alignment-driven boundaries improve success on composite instructions over the best fixed-horizon systems. They must also match or beat a termination head trained without language. That second comparison matters most: if language adds nothing over a plain learned termination predictor, the cross-modal half of this proposal is unnecessary.
Relation to Prior Work
The closest systems all fix the duration and leave the gap unexamined. LISA quantizes language-and-observation features into discrete skill codes and holds each code for exactly $H$ steps. Its authors list $H$ and codebook size as manually tuned. LADS learns a discrete skill codebook from pre-sliced, fixed-length trajectory segments and adds a CLIP-style contrastive loss tying plans to instructions. It never asks how the slicing interacts with that loss. SkillDiffuser conditions a trajectory diffusion model on discrete skills over fixed planning windows. CHD couples high- and low-level diffusion planners with a shared guidance classifier, but it has no language. Its authors name automatic boundary discovery and variable-length segments as future work. We keep the discrete codebook and decoupled training from LADS, because LADS shows they avoid index collapse and the averaging of distinct behaviors. What we change is where segments begin and end, and how the language signal is calibrated.
Several papers motivate the premise that fixed windows are wrong. Learning Navigation Subroutines from Egocentric Videos found that fixed windows cut across natural behavior boundaries. MLSH and goal embeddings via self-play argue that different motor skills need different durations. None of them uses language to find the boundaries.
The gap literature supplies both the measurement and a constraint on the design. Mind the Gap shows that deep encoders squeeze each modality into a narrow cone and that contrastive training keeps the modalities apart. It also shows the gap can be tuned by shifting modality centroids, and that closing it fully is not always best. The consequence for us is that raw cross-modal cosine scores are compressed and poorly calibrated. Any stopping rule built on an absolute threshold of such a score is suspect. We therefore use relative change in a centroid-corrected score, not an absolute threshold. Extending Multi-modal Contrastive Representations reports that the gap inside a pretrained space is largely linear and is best bridged with a linear map. This constrains us to a linear correction and away from a deep projector that could distort the trajectory geometry. ReViSE showed that matching whole distributions with maximum mean discrepancy (MMD, a kernel distance between two sample sets) beats pointwise pair matching on static zero-shot recognition. We test it as an alternative correction.
Language Guided Skill Discovery avoids the gap altogether by turning states into text through handcrafted rules and an LLM. Its authors call that translator a scaling bottleneck. Our approach needs no such translator.
Research Program
The first stage is a diagnostic. We train a trajectory encoder $g$ and use a frozen sentence encoder $e$ on LOReL's 50,000 annotated play trajectories. Following Liang et al., we define the gap as $\Delta = \bar{e} - \bar{g}$, the difference between the mean language embedding and the mean trajectory embedding. We measure $\Delta$ and trajectory-to-instruction retrieval accuracy as functions of how segments are cut. The cutting schemes are fixed lengths $H \in \{5, 10, 20, 50\}$, oracle boundaries, and random variable lengths. On Kitchen, oracle boundaries can be read from the recorded sub-task completion events. On LOReL we approximate them with the success detectors used for evaluation. The prediction is that fixed slicing inflates both the gap and the within-code spread of trajectory embeddings, relative to oracle cuts at matched average length. If it does not, the rest of the program loses its motivation, and we will say so.
The second stage is the stopping rule. For a segment starting at $t_0$ and paired with language description $\ell$, we compute a centroid-corrected alignment score
$$s(t) = \cos\big(g(\tau_{t_0:t}) - \bar{g},\; e(\ell) - \bar{e}\big),$$
where $\tau_{t_0:t}$ is the trajectory from $t_0$ to $t$, $\bar{g}$ is the mean trajectory embedding, and $\bar{e}$ is the mean language embedding. The segment ends at the first $t$ after which $s$ has not improved for $k$ steps. $k$ is a small patience constant, which we report sensitivity to.
During training this rule re-segments the offline data. The LADS-style codebook is then learned on the variable-length segments. During execution the same rule runs online: the current skill code stands in for $\ell$ through its learned language anchor, and the high-level model re-selects when $s$ saturates. A hard cap on duration guards against skills that never saturate.
A second variant replaces centroid subtraction with an MMD penalty in the style of ReViSE. This tests whether distribution-level alignment gives a better-shaped score. To keep the ingredient isolated, we deliberately keep the low-level policy and the codebook identical to LADS.
Baselines
The naive reference is a flat language-conditioned policy with no skills. LISA reports flat baselines at 15% success on image-based LOReL, against 40% for LISA. LISA itself is the fixed-horizon reference. We give it a grid search over $H$ with the same tuning budget our method gets for $k$, so that no one can say we tuned only our own method.
The strongest existing competitors are these. SkillDiffuser reports 43% average success on LOReL Sawyer and 25.21% on complex multi-step tasks, and 23.3% on Meta-World MT10. LADS reports 24.0% on LOReL composite instructions with image observations. LADS is the most direct comparison because our method shares its codebook and policy. Since these numbers come from different evaluation setups, we rerun LADS and SkillDiffuser from their released descriptions on identical data splits, observation types, seeds and compute. We report our reruns alongside the published figures.
The key ablation is a learned termination head of the kind common in the options literature. It is a small classifier that predicts from the state alone whether to stop, trained with the same data and the same boundary supervision budget, and it has no access to language. We add a language-free change-point segmenter on actions and states, and a version of our rule without the centroid correction. We also include an oracle-boundary upper bound.
Beating the field means three things. First, success on composite instructions above our reruns of LADS and SkillDiffuser, with non-overlapping confidence intervals across seeds. Second, higher boundary accuracy against oracle cuts than the language-free termination head. Third, fewer high-level decisions per episode than LISA at its best $H$.
Why not just…?
Why not just grid-search a better fixed $H$? Grid search finds the one horizon that is least bad on average. Kitchen and LOReL mix brief actions like handle turns with long reaches, which is why MLSH and the self-play goal-embedding work argue that no single timescale fits. Our LISA baseline gets the full grid anyway. If its best $H$ matches our success rate and decision count, the claim fails.
Why not just learn a termination head from states, without language? This is the real threat. On a benchmark where sub-task completion is visually obvious, a state-based stopping classifier might do everything our rule does. We make it a matched ablation with equal supervision. We expect language to help most on held-out composite instructions, where the stopping point depends on what was asked and not only on what the scene looks like. For example, "open drawer" stops at a different state than "open drawer and move mug." If the gap between the two methods vanishes on those splits, we will conclude that the cross-modal signal is unnecessary.
Why not just close the modality gap and keep fixed segments? Liang et al. show the gap can be shifted by moving centroids. Zhang et al. show a linear map bridges much of it, and Zhao et al. show that tuning only normalization layers (Tuning LayerNorm in Attention) handles cross-modal shift cheaply. We therefore run LADS with the same centroid correction and fixed segments. If that alone recovers most of the gain, the gain is from calibration and not from variable duration. The first-stage diagnostic, which compares oracle and fixed cuts at matched length, separates these two effects directly.
Why not just translate states to text? LGSD's own authors identify its per-environment handcrafted translator as the bottleneck. Our rule needs only the instructions the dataset already has.
Proposed Experiments
The decisive checks run on LOReL, using both state and image observations and both single-task and composite instructions. Franka Kitchen is used for boundary quality, because its recorded sub-task completions give ground-truth cuts. Meta-World MT10 is a transfer check against SkillDiffuser's reported 23.3%.
We measure five things. The first is task success. The second is boundary F1 against oracle cuts within a tolerance window. The third is the number of high-level re-selections per episode. The fourth is codebook usage entropy, as a check for index collapse. The fifth is the gap magnitude $\|\Delta\|$ together with trajectory-to-instruction retrieval accuracy.
Three results would settle the question. The diagnostic must show that fixed slicing widens the gap relative to oracle cuts at matched length. The alignment rule must beat the language-free termination head on held-out composite instructions. And removing the centroid correction must degrade boundary F1, which would confirm that the cone effect Liang et al. describe actually matters for a stopping rule. If the rule matches the oracle-boundary upper bound within noise, we will report that label-free segmentation is essentially solved on these benchmarks. If it falls well short, the gap between the rule and the oracle bounds how much better alignment could still buy.
References
- LISA: Learning Interpretable Skill Abstractions from Language
- Discrete Latent Plans via Semantic Skill Abstractions
- SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution
- CHD: Coupled Hierarchical Diffusion for Long-Horizon Tasks
- Learning Navigation Subroutines from Egocentric Videos
- META LEARNING SHARED HIERARCHIES
- Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning.
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Extending Multi-modal Contrastive Representations
- Learning Robust Visual-Semantic Embeddings
- Language Guided Skill Discovery
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning