Trusting a Robot's Progress Critic Only Where It Is Self-Consistent

Robots trained with reinforcement learning (RL, learning by trial and error from a reward signal) rarely get useful feedback for free. The true signal is usually one bit at the end of an episode: did the task succeed. A recent fix is to ask a large vision-language model to watch the robot and score how much progress it made between two camera frames. This turns one bit into a dense reward without anyone hand-writing a reward formula. The catch is that these models misperceive the physical world in systematic ways. They mistake a gripper hovering in front of a cup for a grasp. They read a slipped object as progress. They reward the same back-and-forth motion every cycle. When the critic is wrong in a structured, context-dependent way, the policy learns to exploit the error.

We propose a reward signal that treats the learned progress critic as an untrusted evaluator and checks it in two ways that need no extra human labels. The first check is internal. A pairwise critic that says frame A→B is +0.3 and B→C is +0.3 should also say A→C is +0.6, and B→A should be −0.3. Wherever these identities fail, the critic is incoherent, and only the coherent part of its judgments can be safely used as reward. The second check is external. The sparse end-of-episode success bit is never corrupted, so it can audit where the critic's optimism was unjustified. The core claim is that the incoherent part of a pairwise critic is exactly the part a policy can farm into reward loops. If we project the critic onto a consistent "progress potential" and down-weight it in visual contexts where success bits contradict it, the method should match the clean critic's sample efficiency while degrading gracefully as the critic becomes corrupted.

Core Claim

A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots (VLAC) trains a model on frame pairs $(o_i, o_{i+\Delta t})$ to predict progress deltas. It uses those deltas as dense reward, lifting real-robot success from 30% to 90% in 200 episodes. VLAC motivates this proposal and also supplies its main component. It assumes time order equals progress and assumes clean observations. It has no defense when either assumption fails. BEAR finds that 54.8% of multimodal-model failures in embodied tasks are perceptual. That makes a misreading critic the expected case, not the exception.

Let $d(o_i, o_j)$ be the critic's predicted progress from frame $o_i$ to $o_j$. On the frames of a batch of trajectories we fit a scalar potential $\Phi$, a learned "how far along is this frame" score:

$$\hat\Phi = \arg\min_{\Phi} \sum_{(i,j)} \big(d(o_i,o_j) - (\Phi(o_j) - \Phi(o_i))\big)^2 .$$

The residual $\rho_{ij} = d(o_i,o_j) - (\hat\Phi(o_j)-\hat\Phi(o_i))$ is the cyclic, non-conservative part of the critic. Summed around any loop of states it is nonzero, so a policy that cycles through those states collects free reward.

The shaped reward we give the agent is

$$r_t = r^{\text{succ}}_t + \lambda\big[\gamma\, g(o_{t+1})\hat\Phi(o_{t+1}) - g(o_t)\hat\Phi(o_t)\big].$$

Here $r^{\text{succ}}_t$ is the sparse success bit, $\gamma$ is the discount, $\lambda$ is a shaping scale, and $g(o) \in [0,1]$ is a learned trust score for the visual context $o$. Because the bracket has the form of potential-based shaping with potential $\Psi = g\hat\Phi$, the classical invariance argument applies: the optimal policy under the sparse reward is unchanged, whatever $\hat\Phi$ and $g$ are. Corruption in the critic can therefore slow learning by misdirecting exploration. It cannot redefine what the agent ultimately optimizes, and it cannot create reward loops. The trust score $g$ addresses the remaining exploration damage. It is trained to predict, from visual embeddings, whether episodes passing through a context ended in failure even though $\hat\Phi$ reported high progress. Local residual magnitude $|\rho|$ serves as a second, label-free input.

Relation to Prior Work

Three lines of work supply pieces, and none combines them.

The first is corruption-robust RL theory, which bounds how much a corrupted reward can hurt learning. Corruption-Tolerant Asynchronous Q-Learning with Near-Optimal Rates uses trimmed means and a shrinking acceptance region to reach $\tilde O(1/\sqrt T + \sqrt\epsilon)$ error in tabular problems. It also proves that $\Omega(\sqrt\epsilon)$ error is unavoidable when an $\epsilon$ fraction of rewards are corrupted. Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption separates corruption damage from exploration difficulty. Both assume corruption looks like outliers in a per-state reward history. That assumption fails for a neural critic whose errors are repeatable for the same visual context. This constraint is why we do not simply trim critic outputs.

The second is trust-modeling under biased evaluators. Learning When to Trust in Contextual Bandits shows that median aggregation fails when evaluators are reliable in some contexts and lie in others. Its CESA-LinUCB recovers sublinear regret by checking 5% of interactions against ground-truth axioms. We borrow its central move: learn where the evaluator is trustworthy, using sparse ground truth. We change two things. The ground truth is the task's own success bit, so no formal axioms need to be written, and the setting is long-horizon visual control rather than linear bandits. Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback similarly down-weights feedback by uncertainty. Our weight comes from audited disagreement rather than epistemic width.

The third is automated reward design for robots. Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning uses CLIP similarity with a hand-chosen baseline prompt. CoRe combines LLM-written code rewards with VLM preference residuals and reaches 99.0% on Meta-World. Adaptive Reward Design for Reinforcement Learning shapes temporal-logic rewards by distance-to-acceptance, but assumes perfectly observed symbols. All of them assume the reward source is honest. RRM shows, for text reward models, that learned rewards latch onto spurious artifacts. That paper motivates our concern but works only in language.

Research Program

We would build the method on top of VLAC's pairwise critic. If the released weights are unusable in our simulators, we would train a smaller pairwise critic on Open-X-Embodiment and BridgeData-v2 following VLAC's recipe.

The work has three parts. The first is the potential projection, solved per batch by sparse least squares over sampled frame pairs within and across trajectories. The second is the trust model $g$, a small head on frozen critic embeddings. It is trained online with a logistic loss: episodes whose final $\hat\Phi$ exceeded a threshold but failed provide negative labels for the contexts visited late in the episode, and successful episodes provide positive labels. Residual magnitude $|\rho|$ is an extra input feature. The third is analysis. We would test whether $|\rho|$ predicts true critic error, using simulator privileged state to compute ground-truth progress. We would also extend the additive-degradation form $\text{Regret}_{\text{clean}} + f(C)$ to a statement of the shape "exploration cost grows with the corrupted mass of $\Psi$ on visited states, while the optimum is untouched."

We study three kinds of corruption. Natural corruption comes from lighting changes, occlusion, and object slip. Non-monotonic tasks include regrasp after a forced drop and cyclic wiping, which break VLAC's time-equals-progress assumption. Injected contextual corruption mirrors the "contextual liar" model: critic deltas are inverted or inflated in a fraction $\epsilon \in \{0, 0.1, 0.2, 0.3\}$ of the workspace or under one lighting condition.

Baselines

The naive reference is PPO and SAC trained on the sparse success bit alone, with the same environment-step budget. VLAC reports that this collapses in real-robot training, and BEHAVIOR-1K reports that end-to-end RL fails on its long tasks. We expect it to be near zero on hard tasks but immune to critic corruption. That immunity makes it the floor that any corrupted-critic method must stay above.

The strongest existing method is raw VLAC delta shaping, run with the identical critic, policy architecture, environment steps, and a matched hyperparameter sweep (same number of seeds and configurations per method). Its reported result is 30%→90% success in 200 real-robot episodes on the AGILE PiPER arm (VLAC). We found no reported result on Lacuna for VLAC under corrupted or non-monotonic conditions, or on ManiSkill2. Beating it means two things. With a clean critic, our success curve should fall within the seed confidence interval of VLAC's. At $\epsilon \geq 0.2$, our final success and area under the learning curve should be significantly higher, and the reward-hacking rate lower.

On Meta-World we also run CoRe, which reports 99.0% average success over seven tasks. We give it the same corrupted VLM feedback channel and the same label budget, to test whether its formal-plus-residual split already absorbs corruption.

Two robust-statistics baselines come from the corruption literature. The first is critic outputs filtered by a per-context trimmed mean with a shrinking acceptance region, in the style of Robust Async-Q, using kNN contexts in embedding space. The second is a median over an ensemble of three critics.

Two ablations isolate the key ingredients. Projection-only sets $g \equiv 1$; it tests whether removing the cyclic part alone suffices. Trust-only keeps raw $d$ with weights $g$ and no projection. A privileged variant gives $g$ access to simulator ground-truth progress on 5% of steps, mirroring CESA-LinUCB's check budget, as an upper bound.

Why not just…?

Why not just use potential-based shaping with a learned potential and skip the trust model? This is a real threat and is our projection-only ablation. Invariance protects the optimum, not the learning speed. A potential that is confidently wrong in one region, for example one that rates hovering as near-grasp, will steer exploration there for the whole finite budget. If projection-only matches the full method under contextual corruption, we would drop $g$ and report the simpler result.

Why not just take the median of several VLM critics? Learning When to Trust in Contextual Bandits shows that median aggregation incurs linear regret, and can do worse than averaging, when evaluators share context-specific bias. BEAR's finding that frontier models share perceptual failure modes suggests critic errors are correlated. The ensemble-median baseline tests this directly in the embodied setting.

Why not just trim outliers in the critic's outputs? Trimmed means reject rare extreme values. A critic that misreads a hover as a grasp does so every time, so its errors form the bulk of the data for that context, not its tail. The trimmed-mean baseline under injected contextual corruption should show this failure. If it does not, the corruption we model is easier than claimed.

Why not just fine-tune the critic on the success bits? This is plausible. We run it as a baseline: LoRA fine-tuning of the critic on the same terminal labels our trust head sees. Our expectation is that a few hundred bits are too little to reshape a multi-billion-parameter model without drift, while a small trust head can use them. The comparison at equal labels settles it.

Why not just train on sparse reward with massive parallel simulation? ManiSkill2 runs at about 2,500 frames per second, so sparse-only training gets that full budget. But VLAC and BEHAVIOR-1K both report collapse or failure without dense feedback on long tasks, and Process vs. Outcome Reward reports needing 18× more data with outcome-only reward.

Proposed Experiments

The main testbed is ManiSkill2. We use PickCube, for which ManiSkill2 reports 91% simulated success, plus peg and plug insertion and pick-and-place. We add our regrasp and wiping variants, scripted with forced slips. Meta-World provides the comparison against CoRe. A subset of BEHAVIOR-1K activities, scored by their BDDL success predicates, tests long horizons. Pending resources, we would add a small real-arm study under lighting shift.

Metrics are success rate against environment steps, area under the learning curve, and final success as a function of $\epsilon$. We also measure a hacking rate: the fraction of episodes with high critic return but zero success. Finally, we measure the AUROC of $|\rho|$ and of $g$ at predicting true critic error from privileged state.

Four checks decide the proposal.

First, with a clean critic, the method must not cost sample efficiency relative to raw VLAC.

Second, as $\epsilon$ rises from 0 to 0.3, raw VLAC, trimmed mean, and ensemble median should lose substantially more final success than our method. Our method should stay above the sparse-only floor at every $\epsilon$.

Third, on the wiping task, raw VLAC should show cycle farming, with reward rising while success stays flat, and the projected potential should eliminate it.

Fourth, residual magnitude should predict true critic error well above chance. If it does not, the label-free half of the idea fails, and the method reduces to outcome-audited trust weighting.