Let Language Pick the Goal, Let Dynamics Measure the Distance

Robots trained with reinforcement learning need a reward, a number that says how well they are doing at each step. Writing that number by hand for every task does not scale. A popular shortcut takes a vision-language model, an encoder that maps images and sentences into one shared vector space, and rewards the robot for how similar its camera image is to the text instruction. This works poorly for language goals, and the reason is geometric. Image vectors and text vectors sit in separate regions of the shared space. A step of physical progress therefore barely moves the image-to-text similarity, and it can even lower it when the robot must first move away from the goal, for example backing off to get around an obstacle or opening a door before reaching inside.

We propose to split the reward into two jobs that fit what these models are actually good at. Language is used once per task, as a classifier: it marks which previously seen visual states satisfy the instruction. Progress toward those states is measured entirely inside the visual modality, as the shortest-path distance on a directed graph of which states can reach which. In two sentences: the cross-modal model only answers "is this the goal?", which the modality gap does little to harm. "How far is the goal?" is answered by a reachability graph built from image trajectories, which respects obstacles, ordering, and irreversible actions and never has to bridge the gap.

Core Claim

Mind the Gap explains the separation between image and text embeddings in contrastive models. Each encoder squeezes its outputs into a narrow cone, and the contrastive loss keeps the two cones apart. The paper also shows that this gap does not hurt zero-shot classification much, and that tuning it can slightly improve accuracy. On CIFAR-10, adjusting the gap moved top-1 accuracy from 0.9013 to 0.9081. That paper motivates our design: cross-modal embeddings are good at discrimination and unreliable as a metric. LIV trains a joint image-text value model on EpicKitchens and uses the inner product between the current image and the instruction as a dense reward. Its authors report that reward curves for language goals are noisier and flatter than for image goals, and that long concatenated instructions cause "task aliasing." Our claim is that most of LIV's language-goal failure is metric error rather than recognition error. If that is true, a directed visual reachability graph whose goal set is selected by LIV should recover most of the gap between language-goal and image-goal performance, with no manual reward terms.

Concretely, let $G$ be the set of graph nodes whose LIV membership score for the instruction exceeds a threshold. Let $d(n, G)$ be the directed shortest-path cost from node $n$ to any node in $G$, where edge costs come from a learned asymmetric reachability network. The shaped reward is

$$r_t = \gamma\,\Phi(s_{t+1}) - \Phi(s_t), \qquad \Phi(s) = -\big(d(\mathrm{node}(s), G) + d_l(s, \mathrm{node}(s))\big),$$

where $\gamma$ is the discount factor, $\mathrm{node}(s)$ is the nearest graph node to observation $s$, and $d_l$ is the local reachability cost from $s$ to that node. Because the reward is potential-based, it changes how fast the agent learns but not which policy is optimal for the underlying goal. The local-plus-global form follows Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping.

Relation to Prior Work

Our method borrows its graph machinery from three works. Mezghani et al. build a directed graph over offline trajectories using an asymmetric reachability network and reward the agent with graph distance. They report that directed edges beat undirected ones by 11% in success on RoboYoga Walker. However, they work on low-dimensional joint-angle states with goals given as state vectors. We move their construction to image embeddings and replace the goal vector with a language-selected goal set. DGMem builds an online topological graph from RGB images and rewards reductions in path distance, reaching 75% success in the Gibson scene "Nicut" versus 7% for plain PPO. Its goals are spatial or object-detector landmarks, not open-vocabulary instructions. G4RL embeds states so that latent distance reflects graph connectivity, but it uses an undirected graph over MuJoCo coordinates. The Laplacian line of work (Machado et al., DCEO) gives structure-aware exploration bonuses that point along directions of environment flow. These bonuses are not goal-directed, so we treat them as a possible exploration add-on rather than as the reward itself.

On the reward-model side, VIP learns a visual embedding from Ego4D video in which embedding distance tracks progress toward a goal image. It reaches a 40% aggregate online-RL success rate on FrankaKitchen, but it needs a goal photo and assumes progress is a symmetric vector distance. LIV adds language at the cost of the noise described above. Neither builds a graph. The closest thing to our idea would be combining these components. No paper in the file uses a language model only for goal-set membership and measures distance on a visual directed graph.

Research Program

The system has three parts, each trained without task-specific reward code.

The first part is a reachability graph over images. We take frames from offline play or demonstration data (the FrankaKitchen relay dataset used by RPL, and scripted plus random rollouts in Meta-World). We embed them with frozen LIV or VIP encoders and train an asymmetric reachability network on temporal ordering within trajectories, as Mezghani et al. do. We then cluster frames into nodes and add directed edges between nodes the network judges to be reachable in a few steps.

The second part is goal grounding. For each instruction, we score every node with LIV's image-text similarity, normalized within each modality, and take the top-scoring nodes as a soft goal set $G$. We measure membership accuracy separately from reward quality, because it is the only place the modality gap enters.

The third part handles multi-stage instructions. We split the instruction into ordered sub-instructions, ground each to its own goal set, and compute $\Phi$ on a layered graph that must pass through the sets in order. This is our direct test of whether LIV's task-aliasing failure disappears when ordering is enforced by the graph rather than by text concatenation.

The central analysis is an error decomposition. For each task we separately measure recognition error (does LIV correctly identify goal frames?) and metric error (does the reward increase monotonically along successful trajectories?). We compute both with LIV's raw inner product and with our graph distance. If our claim is right, recognition error stays moderate and roughly unchanged, while metric error falls sharply under the graph.

Baselines

The do-nothing reference is the sparse task-success indicator with the same RL agent. It avoids reward hacking but should learn slowly on long-horizon kitchen tasks. It anchors how much any shaping helps.

The strongest existing method for language-specified rewards without manual engineering is LIV's dense reward, the image-text inner product. We use the released LIV encoder unchanged. We also include an in-domain fine-tuned LIV trained on the same offline frames our graph uses, because LIV reports that domain adaptation of CLIP improved real-robot success by over 40% relative to standard fine-tuning. Giving LIV the same data removes the objection that our graph simply sees more in-domain data. No reported result found on Lacuna for LIV's FrankaKitchen or Meta-World success rates, so we rerun it ourselves.

As a goal-image ceiling we run VIP with a true goal image. VIP's reported 40% aggregate online-RL success on FrankaKitchen (VIP paper) is the published reference point, and we reproduce it under our agent and seeds before comparing. As a privileged-state topological baseline we run Mezghani et al.'s graph reward on simulator states, which bounds what the graph can do when perception is perfect.

Two ablations isolate the key ingredient. In the first, the graph is undirected, which tests whether asymmetry matters on visual manifolds as it did on Walker. In the second, the graph is replaced by straight embedding distance to the same LIV-selected goal set, which tests whether topology, rather than the goal-set step, carries the gain.

All methods share the same frozen encoder backbone, the same offline frames, the same model-free online RL agent and replay settings as the VIP online protocol, the same environment-step budget, and an equal hyperparameter-search budget (the same number of configurations per method, five seeds each). Beating LIV means higher mean success at matched steps, with seed-level confidence intervals that do not overlap, on the multi-stage and detour subsets. It also means closing at least half the gap between LIV and the VIP goal-image ceiling.

Why not just…?

Why not just close the modality gap and keep cosine similarity? Liang et al. show the gap can be shifted with a simple mean translation of embeddings. If the noise in LIV's reward came from the gap's offset alone, shifting would fix it cheaply. We will run this as a baseline: LIV embeddings with the per-modality mean subtracted, then cosine reward. This is a real threat on short single-stage tasks. It cannot help on detour or ordered tasks, because a straight similarity of any calibration has no notion of what must be done first. The detour and multi-stage subsets settle the question.

Why not just retrieve a goal image with language and hand it to VIP? This is the most serious alternative. It also uses language only for recognition and keeps distance within the visual modality, so it shares our core insight minus the graph. VIP distance is symmetric and Euclidean in latent space, and VIP's own results show weaker success on harder configurations. Our second ablation is exactly this pipeline: the same LIV-selected goal set, embedding distance instead of graph distance. If it matches the full method, we will report that the topology is unnecessary and that the contribution reduces to "use language as a classifier." That would still be a useful finding, but a smaller one.

Why not just have an LLM write the reward code? Programmatic rewards need simulator coordinates. Our setting assumes only camera images, where those coordinates do not exist. Hand-written or generated proxy combinations are also prone to exploitation, as Active Reward Learning documents. We include a privileged code-reward run in simulation only, as a reference for what state access buys. We do not treat it as a competitor in the image-only regime.

Proposed Experiments

We evaluate on FrankaKitchen, using single-stage tasks, the four-step sequences where RPL averaged 2.4 completed steps, and Meta-World tasks with language instructions. We add a detour subset in both suites, made of task variants where states most similar to the instruction are not on the shortest physical path, such as reaching an object behind a closed cabinet door. On these, greedy similarity should stall. Metrics are success rate, number of stages completed for sequences, environment steps to a fixed success level, and two reward diagnostics: the Spearman correlation between reward and time along held-out successful demonstrations, and goal-set membership precision and recall against simulator success labels.

The first decisive check is the error decomposition. If graph rewards raise demonstration monotonicity well above LIV's, while membership precision stays comparable to LIV's own recognition accuracy, then the failure LIV reports was metric failure, which is our claim. The second is the retrieve-then-VIP ablation on detour tasks, which determines whether the graph matters. The third is directed versus undirected edges on irreversible kitchen actions, such as a switch that flips one way. This tests whether Mezghani et al.'s asymmetry result carries over from joint angles to pixels.

The main failure mode we watch for is graph drift. Nodes merge visually similar but physically different states, for example a drawer barely open versus closed. We report how success degrades as cluster resolution coarsens, so the method's dependence on the visual embedding is measured rather than assumed.