Grounding the progress critic: object-level dense rewards for language-conditioned robot RL

Robots trained with reinforcement learning on long tasks usually get a single success-or-failure signal at the end of each attempt. That is too little feedback to learn from. A growing line of work replaces it with a learned "progress critic": a model that looks at two camera frames and an instruction and says how much closer the robot got to finishing. These critics already work. A pair-wise progress critic lifted real-robot RL success from 30% to 90% in 200 episodes (VLAC), and a time-ordered value model gave the best CALVIN generalization result to date (RFTF). Both judge whole, downsampled frames, though. Neither checks which object the robot is acting on. A robot that makes a demonstration-like reach toward the blue block when told "push the red block" looks like progress to them, and RL will optimize exactly that mistake.

We propose to make the critic answer the right question by grounding it at the level of objects. The instruction's noun phrases are matched to per-object tokens in each frame, and progress is predicted from how those specific objects and their relations change. The critic is trained with temporal foils: the same demonstration video paired with an instruction whose target object has been swapped for another object visible in the scene. These foils cost nothing to generate. A foiled instruction should receive no progress from a motion that serves the original one. The idea transfers the single-word-mismatch test of FOIL it! from static captions to videos used as reward. The claim is narrow and testable. When distractors are present, a grounded, foil-trained critic gives RL rewards that fail less often by manipulating the wrong object. A global critic trained on the same foils cannot match it, and the gap widens as distractors become more similar to the target.

Where this sits

The three directions play different roles here. Fine-grained cross-modal alignment, meaning matching individual words to individual image regions instead of whole captions to whole images, supplies the mechanism. Temporal integration is the input regime, since the critic must compare frames in causal order. Sparse reward is the problem being solved.

Three papers motivate the gap. VLAC trains an InternVL2 actor-critic on about 40M instances to predict progress deltas between frame pairs. Its frames are global tokens, and the research file notes it can reward motion toward the wrong target if the trajectory resembles expert progress. RFTF trains a value model by temporal contrastive ranking on successful videos and feeds it into PPO, again from global image features. Robot Policy Learning with Temporal Optimal Transport Reward matches rollout frames to expert frames under a temporal mask, using pooled visual context embeddings with no language at all. We keep their pair-wise, time-aware formulation. We change what the critic looks at and what counterfactuals it is trained against.

Two papers supply the method. ObjEmbed gives each detected object a semantic token and a box-quality token in one forward pass, reaching 89.5% average accuracy on RefCOCO/+/g. We use this style of per-object token as the critic's visual unit. Long-VLA shows that injecting Grounding DINO object features into a policy helps long-horizon imitation, reaching 4.75 tasks on L-CALVIN. It uses grounding only inside a behavior-cloned policy, never inside a reward. Symmetrical Visual Contrastive Optimization supports the training principle: minimal contrasts force token-level binding. We apply that principle on the language side of a video-reward model.

Two papers constrain the design. CuRPO and Video-OPD both find that long chain-of-thought reasoning degrades localization, in spatial and temporal grounding respectively. So the critic predicts its progress delta directly, with no generated reasoning. The process-versus-outcome comparison in agentic retrieval (Process vs. Outcome Reward) reports 18x data savings from dense step rewards. It reminds us that dense rewards help only when they track the right quantity, which is the point at issue.

Core Claim

The proposed critic $\Delta p_\theta(o_i, o_j, \ell)$ takes two observations $o_i, o_j$ (with $i<j$) and an instruction $\ell$, and outputs a progress change.

The instruction's entity phrases attend over per-object tokens in each frame. The critic reads the change in the attended objects' tokens and in their relation to the gripper and to any goal location, plus one global token for context.

Training combines three signals. The first is time-ordered demonstration pairs regressed to their normalized progress gap, as in VLAC. The second is reversed pairs, labeled with the negative gap. The third is a margin loss against a foil instruction $\tilde\ell$ in which the target entity is swapped for another object present in the frame:

$$ \mathcal{L} = \big(\Delta p_\theta(o_i,o_j,\ell) - (\tfrac{j-i}{T})\big)^2 + \lambda\,\max\!\big(0,\; m - \Delta p_\theta(o_i,o_j,\ell) + \Delta p_\theta(o_i,o_j,\tilde\ell)\big) + \mu\,\Delta p_\theta(o_i,o_j,\tilde\ell)^2 . $$

Here $T$ is the demonstration length, $m$ is a margin, and $\lambda,\mu$ weight the foil terms. The last term pushes foil progress toward zero. Because foils reuse existing demonstrations, the foil data is as large as the demonstration set at no collection cost. An auxiliary box loss keeps the phrase-to-object attention tied to detector boxes.

Research Program

The work has three stages. Each stage can fail on its own, which is informative.

First, we build a critic-level diagnostic, a temporal FOIL test. Frame pairs come from CALVIN demonstrations, which naturally contain several colored blocks and color-specific instructions. Each pair is scored under its true instruction and under a foil. We report how often the true instruction receives more progress than the foil. Pairs are bucketed by distractor similarity (different shape, same shape with different color) and by the target's pixel size. This stage answers whether global critics actually fail this test before any RL is run. If they do not fail, the project's premise is wrong and we stop.

Second, the critic replaces RFTF's value model inside RFTF's PPO recipe, on the same policy backbones (Seer-Large and GR-MG). We use standard CALVIN and a distractor variant: added same-shape, different-color blocks, with instructions naming one of them.

Third, we probe a secondary weakness both critics share. The research file notes VLAC and RFTF assume progress rises with time, which breaks on recovery. Grounding gives a principled fix: if progress is read from object states rather than elapsed time, a knocked-over block should lower progress and re-righting it should raise it. We test this by splicing failure prefixes onto success suffixes. The critic is not trained on these splices; we check whether the grounded critic tracks the drop and recovery. This is a stretch result, not a core claim.

Baselines

The naive reference point is PPO with the sparse terminal success signal only, using RFTF's training recipe. RFTF reports this ablation directly: sparse reward drops Seer-Large from 4.296 to 4.225 average success length on CALVIN ABC→D (RFTF). That small margin is itself informative. On the standard benchmark, which has few confusable distractors, dense rewards add little, so standard CALVIN alone cannot separate our critic from RFTF's.

The strongest existing method on this exact benchmark is RFTF with Seer-Large, reporting 4.296 average success length on ABC→D and 4.301 with Env-D adaptation. We rerun it with the same backbone, environment steps, PPO hyperparameters, and tuning budget. Only the reward model changes, and both reward models are trained on the same ABC demonstrations. On standard CALVIN, beating it means matching 4.296 within seed noise; we do not expect a large gain there. On the distractor variant, beating it means a higher average success length together with a lower rate of failed episodes in which a non-instructed block was moved. We will report seed spread and claim a win only if it exceeds that spread.

We also run VLAC's pair-wise critic as a reward, if the released checkpoint permits it, fine-tuned on the same CALVIN demonstrations under the same budget. No reported result found on Lacuna for VLAC on CALVIN. TemporalOT is a cross-check on Meta-world, where it reports 61.1% average success over 9 tasks with two demonstrations, versus 35.9% for ADS (TemporalOT). We do not expect to beat it in that language-free, single-object setting. We report it to show the grounded critic does not regress there.

Two ablations isolate the key ingredient. The first trains RFTF's global value model with exactly our foil and reversed-pair losses, which separates "foil data" from "object grounding." The second trains our grounded critic without foils, which separates "grounding" from "foil data." The core claim survives only if the grounded, foil-trained critic beats both on the distractor variant.

Why not just…?

Why not just train the existing global critic on the foils? This is the most serious threat. It is exactly the first ablation above. Foils share frames with the true pairs, so a global critic is forced to use language-image interaction and might learn enough. Our prediction is that it fails when targets are small and distractors differ only in color, because pooled features at downsampled resolution blur small objects. The similarity- and size-bucketed temporal FOIL test will show whether that is true. If the global critic matches ours in every bucket, the contribution becomes the foil training recipe, not the architecture, and we will write it up that way.

Why not just feed Grounding DINO features into the value model, as Long-VLA does for its policy? That is a cheaper form of grounding: features are added, but phrases are never bound to objects. Long-VLA shows the injection helps imitation. Nothing in the file shows that injected features make a reward model sensitive to which named object moved. We include this variant, global critic plus FiLM-injected detector features with foil training, as an additional arm. If it matches the full model, we adopt it as the simpler design.

Why not just use a large vision-language model as a success detector plus hindsight relabeling? Relabeling, as in HAL, turns failures into successes for other instructions. That is useful, but it still yields one signal per episode, not per step. The RFTF ablation shows that on CALVIN the gap between sparse and dense reward is small to begin with. We will run sparse reward with relabeling as an extra baseline. If it equals our critic on the distractor variant, the dense grounded reward is not earning its cost.

Why not just hand-write an object-distance reward from a detector? On CALVIN, the gripper-to-target distance is available, and this is a real oracle-style competitor for reaching tasks. It must be rewritten for every task type, though (sliding, opening, stacking). It is also prone to the oscillate-near-the-object hacking the research file describes, and Learning Dense Reward with Temporal Variant Self-Supervision reports learned progress beating Euclidean-distance rewards on door opening. We include it as an upper reference on reach-dominated tasks only.

Decisive Checks

The project rests on three results. The first must hold before anything else is run. Global critics must rank true over foil instructions clearly worse than the grounded critic in the hard buckets of the temporal FOIL test. The second is the main result: on the CALVIN distractor variant, PPO with the grounded critic must lower the wrong-object failure rate and raise average success length over RFTF. This must hold under matched compute and against the global-critic-plus-foils ablation. The third guards against regressions: on standard CALVIN ABC→D the grounded critic must stay within noise of RFTF's 4.296.

If the first check fails, global critics are not object-blind in practice and the project ends early and cheaply. If the second succeeds only through foil data, the finding is a training recipe for any critic, which is still worth reporting but smaller. The recovery-splice probe tests whether object-state progress also removes the reliance on progress rising with time. It is reported only as a secondary result.