Robots learning by trial and error need a reward signal, a number saying how close they are to done. Hand-writing that number for every task does not scale, so two automated shortcuts have emerged. The first uses visual embeddings trained on video. The best of these produce smooth, steadily rising rewards, but only if someone supplies a photo of the finished scene. The second uses vision-language models like CLIP, which let you state the goal in words, but their rewards are bumpy, and they are distorted by a known geometric quirk: in these models, image and text embeddings sit in separate regions of the space. We propose to keep the smooth video-trained reward and replace the goal photo with a sentence. We do this by learning a small linear map that sends a CLIP text embedding of a goal description to the point where the finished-scene image would have landed in the video-trained space.
The bet rests on one empirical finding: the offset between modalities inside a contrastive model is close to linear and can be bridged with a single linear layer, while nonlinear layers damage the geometry. If that holds for robot goals, a few hundred captioned final frames should be enough to make text-specified rewards nearly as well-ordered as rewards computed from a true goal image. The stronger version of the claim is that this works for task descriptions never seen during training. That stronger claim is what separates the method from simply looking up a stored photo.
Core Claim
Value-Implicit Pre-Training, or VIP, learns an image embedding from human video. In that space, negative Euclidean distance to a goal frame acts as a smooth, monotonic dense reward. It is a strong reward, but it cannot accept language, and the authors list goal-image dependency as its main limitation. ZeST does accept text goals. It computes CLIP distances between observations and goal specifications, and it subtracts the initial-frame embedding ("delta features") to suppress background differences. However, it never corrects the text-image offset. Mind the Gap documents that offset and shows that shifting it changes zero-shot accuracy. Ex-MCR reports that the offset inside a model is linear.
Our claim is that a linear map from frozen CLIP text embeddings into frozen VIP space can serve as a goal "embedding generator." To make that work, the map must be trained so that the reward rises along successful trajectories, not merely so that text and image land close together. Concretely, let $t$ be the CLIP text embedding of a goal description and $\phi(s)$ the VIP embedding of frame $s$. We learn $W, b$ so that the predicted goal $\hat g = Wt + b$ defines the reward $r(s) = -\|\phi(s) - \hat g\|_2$. The training loss is
$$\mathcal{L}(W,b) = \|\hat g - \phi(s_T)\|_2^2 + \lambda \sum_{i<j} \max\big(0,\ \epsilon + r(s_i) - r(s_j)\big),$$
where $s_T$ is the final frame of a successful captioned trajectory, $s_i, s_j$ are frames sampled from that trajectory with $i$ earlier than $j$, $\epsilon$ is a margin, and $\lambda$ weights the ordering term. The first term regresses onto the goal location. The second term asks directly for the property a planner needs, which is that reward increases over time.
The comparison we care about is between this projection, the real goal image, and two cheap substitutes for it: a retrieved goal image and a mean-shift gap correction.
Research Program
The work has three parts.
Measuring the offset. We first ask how much of the error in CLIP text-image rewards is due to the offset at all. Along expert FrankaKitchen trajectories, we split each image-to-text distance into three pieces: a constant shift, which the mean-shift correction from Mind the Gap removes; a static offset fixed for each task; and a residual that varies over time. We then measure, with Kendall's rank correlation, how well each corrected reward orders frames by time. If simply removing the mean offset already gives VIP-level ordering, the projector is unnecessary, and we would report that as the finding.
Training the projector. We train $W, b$ on captioned final frames and trajectories drawn from a subset of FrankaKitchen tasks. We compare a linear map with an MLP, following Ex-MCR's warning that nonlinear projectors disrupt geometry.
How the goal is worded. We borrow a lesson from text classification. Chai et al. found that replacing class indices with descriptive label text helps classifiers. By analogy, we contrast two phrasings. Instructions ("open the microwave") describe actions. State descriptions ("the microwave door is open, the kettle is on the left burner") describe how the finished scene looks. We expect state descriptions to project better, because the target $\phi(s_T)$ encodes a scene, not an action.
Baselines
The naive reference is sparse success reward, which is zero until the task is done. In MPPI planning it supplies no gradient. We also run it in online RL so the dense rewards have a floor to beat. In the planning comparisons below, MPPI means sampling-based trajectory optimization, which is the setting where VIP reports its strongest reward-only results.
The direct language competitor is CLIP cosine similarity to the goal text, with and without ZeST's delta features. It is run on the same frames, prompts, and MPPI budget. ZeST reports 90.77% of expert performance with offline Decision Transformer. VIP reports that CLIP rewards "failed completely" on Hard FrankaKitchen variants under MPPI. Beating this baseline means a nonzero solve rate on Hard variants and higher success on the rest.
The gap-correction baseline is CLIP after subtracting the mean image-text offset, as in Mind the Gap, estimated from the same captioned data our projector sees. This isolates whether a constant shift is enough.
The upper reference is VIP with the true goal image. VIP reports roughly 30–45% of FrankaKitchen task configurations solved with MPPI and a 40% aggregate success rate in online RL. Matching this figure within a few points without a goal image is the target. Matching it on held-out task descriptions is the decisive result.
The strongest practical competitor is VIP with a retrieved goal image. Here we encode the text with CLIP, find the nearest captioned final frame in the same training pool, and use that frame as VIP's goal. It gets exactly the projector's data and no extra tuning.
Ablations remove the ordering term (setting $\lambda = 0$), replace the linear map with an MLP, and swap state descriptions for instructions. Every method gets the same camera views, the same MPPI sample budget, and the same number of hyperparameter trials on held-in tasks only.
Why not just…?
Why not just retrieve a goal image from the caption and use VIP? This is the real threat. On tasks whose final frames are in the pool, retrieval may tie the projector or win. Our claim is therefore limited to two settings: held-out descriptions, and compositional goals whose exact final configuration was never photographed, such as kettle and microwave states combined differently. Retrieval can only return a stored scene, while the projector can place a goal between stored scenes. If retrieval matches the projector on held-out compositions too, the projector adds nothing, and we will say so.
Why not just subtract the mean modality gap from CLIP? Mind the Gap shows that shifting the offset moves zero-shot CIFAR10 top-1 accuracy only from 0.9013 to 0.9081. That suggests a constant shift fixes calibration, not ordering. CLIP's bumpy reward landscape, documented in VIP, is a problem with the ordering of frames over time, and a constant shift cannot repair that. The diagnostic in the research program tests this directly rather than assuming it.
Why not just fine-tune CLIP on robot video with language, as R3M does? R3M trains visual features with video-language alignment. Its reported gains, such as 56% versus CLIP's 24% on real apartment tasks, come from imitation learning, not from rewards driven by distance to a goal. No reported result found on Lacuna for R3M's language head used as a planning reward on FrankaKitchen. We include R3M's language-alignment score as a secondary baseline, under the same data and tuning budget as the other methods. Fine-tuning also costs far more than fitting a linear map and risks eroding CLIP's text semantics, which held-out descriptions rely on.
Proposed Experiments
The main benchmark is FrankaKitchen using VIP's 36 task configurations, with tasks split by appliance so that test descriptions name held-out target states. MPPI planning uses VIP's reported protocol. Online RL uses the same agent for every reward.
Reward quality is also measured without any policy, using Kendall's $\tau_a$ against time on expert trajectories. This is the metric RynnValue reports on RBM-EVAL-OOD (0.675 for RynnValue-8B). We use the metric but do not claim comparability to that model, which conditions on language inside a large multimodal LLM rather than through distance.
The decisive checks are three:
- Held-out descriptions. Does the linear projector, trained with the ordering term, come within five points of VIP-with-goal-image success on held-out descriptions while clearly beating retrieval there?
- Ordering term. Does removing the ordering term collapse $\tau_a$ toward the mean-shift baseline?
- Phrasing. Do state descriptions beat instructions?
A secondary stress test probes the known limits of distance rewards: occlusion by the robot arm, reported as a ZeST failure mode, and directional tasks such as closing versus opening, where symmetric distance can mislead per Mezghani et al.. These are reported as failure analyses rather than claimed fixes.
References
- VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training
- Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Extending Multi-modal Contrastive Representations
- Description Based Text Classification with Reinforcement Learning
- R3M: A Universal Visual Representation for Robot Manipulation
- RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance
- Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping