Offline reinforcement learning (RL) trains a control policy from a fixed log of past behavior, with no new interaction. For goal-reaching tasks, the most reliable finding is counterintuitive. Broad, noisy, low-quality logs beat clean expert ones, because the agent has to "stitch" fragments of many trajectories into routes nobody demonstrated. Cross-modal learning, the business of putting images and text into one shared embedding space, tends to show the opposite. Noisy or mismatched image–text pairs corrupt the shared space, and even clean training leaves a persistent geometric separation between the two modalities, called the modality gap. A robot told "put the cube in the drawer" while seeing camera pixels sits on both sides of this tension at once. It needs messy data to learn where it can go, and clean data to learn what the words mean.
This note proposes that the two needs be served by different parts of the model, fed by different slices of the data. The model learns a reachability geometry, meaning how far each image-state is from each other image-state in steps, inside the observation modality alone, using every transition in the log however poor. The modality gap is crossed in exactly one place: a small model that maps an instruction to a set of points in that geometry, trained only on the trustworthy instruction-labelled subset. The claim is empirical and falsifiable. When language-conditioned offline RL fails on low-quality data, the damage is concentrated where text meets state, not in the reachability estimate. Routing data by quality to the matching component should therefore make the agent about as robust to poor control data as a state-goal agent, while confining its sensitivity to bad labels to a single measurable grounding error.
Core Claim
The closest existing work is Latent Representation Alignment for Offline Goal-Conditioned RL (LAVL). LAVL writes the goal-conditioned value as a negative distance between two different learned embeddings, $V(s,g) = -\|\phi_S(s)-\phi_G(g)\|_2$. Here $\phi_S$ embeds the current state and $\phi_G$ embeds the goal. This structure stops values from "leaking" through walls, and LAVL loses only 1.1% when moving from full to fragmented stitch data, where baselines lose 20–30%. LAVL's states and goals are always the same kind of low-dimensional vector, so it never has to cross a modality gap. The obvious extension is to make $\phi_G$ a text encoder and train end to end. We argue that this extension mixes two problems with opposite data needs into one objective.
What we add is not a new value architecture. It is a decomposition with a data-routing rule. Reachability is learned within one modality, from all data. Grounding is a separate set-valued map, learned from clean pairs only. We also add a diagnostic showing why the direct cross-modal distance is fragile.
Why the Gap Bites a Distance-Based Value
Mind the Gap shows that contrastive image and text embeddings sit in separate cones, offset by a roughly constant vector. Is the Modality Gap a Bug or a Feature? shows that this offset $c$ is orthogonal to the subspaces in which each modality varies. Suppose a distance value is computed directly between a frozen image embedding and a frozen text embedding. Write the within-subspace difference as $a$, so the value becomes
$$V(s,g) = -\sqrt{\|a\|^2 + \|c\|^2}, \qquad \frac{\partial V}{\partial \|a\|} = -\frac{\|a\|}{\sqrt{\|a\|^2+\|c\|^2}}.$$
The orthogonality removes the cross term. The second expression says that the value's sensitivity to actual progress shrinks toward zero as the agent nears the goal ($\|a\| \ll \|c\|$). Noise in value estimates from low-quality data stays the same size, so the signal-to-noise ratio of the final approach falls roughly in proportion to $\|a\|/\|c\|$.
If $\phi_S$ is trainable, it can learn to absorb $c$. But the shift that does so can only be learned from instruction-labelled transitions, which is exactly where label noise and scarcity sit. Either way, the gap has to be crossed somewhere, and wherever it is crossed inherits the quality of the paired data. This motivates the design and gives a measurable prediction: failures of direct cross-modal value functions under noisy data should cluster in the last few steps before the goal.
Research Program
The system has three learned parts.
The first is an asymmetric reachability model in the observation modality, following LAVL's distance form and continuity regularizer. It is trained on image observations with hindsight goals, meaning future frames of the same trajectory relabelled as goals. Because it never sees language, it can use explore and stitch data without semantic risk.
The second is a grounding map from an instruction to $K$ goal points in the $\phi_G$ space, with $K$ small, for example 4. One sentence such as "go to the north-east room" corresponds to many states. This granularity mismatch is identified in Learning Robust Visual-Semantic Embeddings, which finds that pairwise contrastive losses align sample points without aligning the underlying manifolds. The value of an instruction $\ell$ is then a soft minimum of distances to the predicted points:
$$V(s,\ell) = -\operatorname{softmin}_{k}\ \|\phi_S(s) - \hat{z}_k(\ell)\|_2,$$
where $\hat{z}_k(\ell)$ is the $k$-th predicted goal point. The grounding map is trained to cover the embeddings of final states of trajectories labelled with $\ell$, and only on pairs that pass a consistency check. That check asks whether the labelled final state is reachable-close to other states carrying the same instruction under the frozen reachability model, which lets the clean geometry police the labels.
The third is LAVL-style hierarchical policy extraction, unchanged, so that any gain is attributable to the decomposition and not the controller.
The analysis component measures three quantities for every method: the modality-gap magnitude $\|c\|$, the near-goal value slope, and the fraction of failures occurring within the last 10% of the path. These test the compression mechanism directly.
Baselines
The naive reference point is language-conditioned goal behavior cloning. This means imitating logged actions given the observation and instruction, in the style of LLaRA. LLaRA turns robot actions into language-model outputs and reaches 90.0% on VIMA-Bench L1 with 12% of the data, but it only imitates. It sets the floor that offline RL must beat on suboptimal data. OGBench reports goal-conditioned behavior cloning at 0% on its hardest environments, so we expect it to collapse on stitch and explore.
The strongest existing competitor is LAVL ported directly to language goals. $\phi_G$ becomes a text encoder followed by a trainable head, and everything is trained end to end on all data. It is the strongest because it is the best reported method on the benchmark family we use, and it differs from our proposal only in where and with what data the gap is crossed. We run it with the same image encoder, the same policy extraction, the same total gradient steps, and the same hyperparameter search budget, which we fix at 20 configurations per method. On state-based OGBench, LAVL reports an 85% success rate on giant mazes against 68% for the next-best method, OTA, and 88% on Scene manipulation (all from LAVL). No reported result was found on Lacuna for any method on language-conditioned or pixel-based variants of these tasks. We therefore rerun LAVL with oracle goal states on our pixel datasets as an upper reference.
A second strong competitor is goal-conditioned implicit Q-learning (one of OGBench's standard baselines) with a language goal encoder. This tests whether a conventional conservative critic sidesteps the issue.
Two ablations isolate our ingredients. The first keeps the decomposition but trains the grounding map on all labelled data, without quality routing. The second keeps routing but uses a single goal point ($K=1$) instead of a set. Beating the end-to-end LAVL port means two things. Under clean labels, we must match it within noise. Under 20–40% label noise or 1% label coverage, we must lose significantly less success, with the remaining loss explained by grounding retrieval error.
Why not just…?
Why not just close the gap post hoc by subtracting the mean offset? This is a genuine threat. If the gap really is a constant orthogonal vector, as Is the Modality Gap a Bug or a Feature? finds, then centering removes the compression term in the equation above, and that paper reports up to 10% robustness gains from this trick at no cost. We include frozen-encoder LAVL with and without centering as a baseline. If centering recovers most of the drop, we report the cheap fix as the main result. What centering cannot fix is the one-sentence-to-many-states mismatch, or corruption of the alignment by mislabelled pairs, which Learning Robust Visual-Semantic Embeddings identifies as a manifold-level, not centroid-level, problem. The label-noise axis of our experiments separates the two.
Why not just filter out low-quality trajectories? OGBench shows that pick-and-place success falls from 99% to 6% when noisy coverage is removed, so control-quality filtering destroys what stitching needs. Our routing filters only the labels used for grounding and keeps every transition for reachability. A "filter everything by label confidence" baseline tests this distinction directly.
Why not just rely on a strong frozen vision-language encoder? Instabilities of Offline RL with Pre-Trained Neural Representation shows that frozen, expressive features can still violate Bellman completeness, meaning the feature space is not closed under the value-update operation. In that work, LSTD error on Walker2d rose from 13.88 to 67.78 when distractor data was added. Our frozen-CLIP-both-sides baseline checks whether the same amplification appears for multimodal features. The compression analysis predicts an additional, gap-specific failure on top of it.
Why not just turn the instruction into a goal image and use a state-goal agent? This is close in spirit to our grounding map. The difference is that we predict a set of latent points rather than one rendered image, and we train them only on routed data. Our $K=1$ ablation approximates the single-goal-image version without the cost of a generator.
Proposed Experiments
We use OGBench's pixel (64×64) maze and cube/scene environments, which have navigate, stitch and explore quality tiers. We add an instruction layer: templated room- and object-level instructions are attached to final states, and paraphrases are held out for testing. Semantic quality is varied independently on two axes: label noise of 0, 20 or 40% swapped instructions, and labelled fraction of 100, 10 or 1% of trajectories. Crossing these with the three control tiers gives a grid that separates control suboptimality from semantic misalignment, a distinction no existing triage metric draws.
The primary metric is success rate over five seeds. Secondary metrics are the gap to the oracle state-goal agent, the near-goal value slope, the final-approach failure fraction, and grounding retrieval accuracy, meaning whether the predicted goal points land closest to correct final states. As a secondary check on transfer to multimodal prompts, we run the instruction-conditioned VIMA-Bench tasks used by LLaRA, comparing against its imitation result.
Three outcomes would decide the question. First, along the control-quality axis at clean labels, our method's degradation from navigate to stitch should track the oracle LAVL's, and the end-to-end port's should be larger. Second, along the label-noise axis, our loss in success should be predicted by grounding retrieval error alone. If reachability-only probes stay flat while success falls, the decomposition is doing its job. Third, frozen cross-modal distance values should show failures concentrated near the goal, and centering should remove them. If the end-to-end port matches our method across the noise grid, or if centering alone closes the whole gap, the central claim is wrong, and we would report it as such.
References
- Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Is the Modality Gap a Bug or a Feature? A Robustness Perspective
- Learning Robust Visual-Semantic Embeddings
- LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
- OGBench: Benchmarking Offline Goal-Conditioned RL
- Instabilities of Offline RL with Pre-Trained Neural Representation