Robots learning from trial and error usually see only one signal: did the task succeed at the end or not. With so little feedback, a learner exploring from scratch almost never succeeds, so it never learns. The current fix is a large pretrained "progress critic", a vision-language model that looks at camera frames and estimates how far along the task is. Its estimate is turned into a dense reward. This works remarkably well when the critic is right. When it is wrong it fails in a systematic way: the critic treats a fumble, a pause, or a repeated wiping motion as progress, and the robot learns to repeat it. We propose to keep the pretrained critic but attach a small trainable correction to it. That correction is updated by an outer optimization loop whose only goal is to make the robot's next policy update raise its real, sparse success rate.
There are two design choices. First, the correction is applied to a potential, a score over states whose differences define the reward. This form provably leaves the best policy unchanged, so the outer loop can only change how fast the robot learns, never what it finally learns. Second, the outer success signal is made nonzero from the first episode by starting some evaluation rollouts from states along the few available human demonstrations, close to the goal. In two sentences: a frozen, pretrained progress model gives dense rewards, and a lightweight adapter on it is meta-trained so that each policy update increases true task success. Demonstration states provide the success signal that this meta-training needs before the robot can succeed on its own.
Core Claim
Robot reward learning has split into two camps that do not use each other's strengths. The first is pretrained progress critics. Robo-Dopamine trains a Qwen2.5-VL process reward model on about 35 million samples and uses it as a fixed potential, taking a real robot from 0% to 95% success in 150 online rollouts. VLAC predicts progress between pairs of frames and lifts real-robot success from 30% to 90% in 200 episodes. Both models are frozen during reinforcement learning (RL). VLAC's authors acknowledge that its assumption that progress always increases with time breaks on mistakes and cyclic tasks.
The second camp is bi-level reward optimization: an inner loop trains the policy on a parameterized reward, and an outer loop adjusts that reward to improve true task return. LIRPG, the self-tuning-network method of Learning Intrinsic Rewards as a Bi-Level Optimization Problem, and BiPaRS all do this, but only on low-dimensional state inputs. They also stall when the outer objective is a 0/1 success that is never achieved, because then its gradient is identically zero.
Our claim is that the combination is better than either camp alone. The pretrained critic supplies a strong prior, which addresses the shortage of robot data. The outer loop repairs exactly those critic errors that mislead the policy, which addresses the failure of hand-designed and frozen rewards. Writing the reward as a learned potential also resolves the open tension in the file between Robo-Dopamine's insistence on policy-invariant shaping and BiPaRS's freely weighted shaping. We restrict learning to the potential, so the invariance guarantee holds throughout meta-training.
Concretely, let $\Phi_0(s)$ be the frozen critic's progress estimate for observation $s$ and $g_\phi(s)$ a small trainable adapter. The adapter is a low-rank correction on the critic's last hidden layer plus a scalar head. The inner policy $\pi_\theta$ is trained on
$$r_\phi(s,a,s') = r_{\text{sparse}}(s,a) + \gamma\,\Phi_\phi(s') - \Phi_\phi(s), \qquad \Phi_\phi(s) = \Phi_0(s) + g_\phi(s),$$
where $\gamma$ is the discount factor and $r_{\text{sparse}}$ is the 0/1 success signal. The outer loop updates $\phi$ by differentiating through one inner update $\theta' = \theta'(\phi)$ to maximize
$$J_{\text{outer}}(\phi) = \mathbb{E}_{s_0 \sim \rho_\kappa}\big[\text{success of } \pi_{\theta'(\phi)} \text{ from } s_0\big],$$
where $\rho_\kappa$ mixes true task start states with states sampled from demonstrations. The mixing weight $\kappa$ shifts toward true start states as success rises.
Research Program
The method borrows from three sources. The inner learner is a demonstration-augmented off-policy actor-critic: RL on $r_\phi$ plus a behavior-cloning term over the demonstrations. The behavior-cloning term follows the inner-loop design of EMRLD, which showed that such a term lets meta-RL escape the zero-reward regime even with noisy, truncated demonstrations. EMRLD meta-learns a policy initialization, not a reward. We transfer its lesson to reward parameters and add the demonstration-reset outer objective, which EMRLD does not need.
The hypergradient is computed with one-step unrolling through the critic target and the actor step, as in LIRPG and BiPaRS. We expect the second-order cost to stay manageable because only the adapter is differentiated and the large critic backbone is frozen. Its features are computed once per observation and cached. The meta-hyperparameters (outer learning rate and adapter size) are a known weak point; LIRPG fell below its baseline on Ant. We therefore treat them as part of every method's tuning budget rather than hiding them.
Our analysis will cover three things. We will plot the learned correction $g_\phi$ against held-out success labels to check where it moves the potential. We expect it to lower the potential on loitering and fumble states and to break false monotonic progress in cyclic segments, which addresses the failure VLAC reports. We will test whether policy invariance holds empirically by comparing final success with and without shaping at convergence. We will also measure wall-clock and memory overhead against the static-critic pipeline.
The critic must be one we can run. If a released Robo-Dopamine or VLAC checkpoint is usable, we use it. Otherwise we pretrain a smaller pairwise progress critic following VLAC's recipe on RoboCasa MimicGen trajectories. Either way the critic is frozen before RL begins.
Baselines
The naive reference point is the same demonstration-augmented actor-critic trained on the sparse reward alone. It receives the same 50 human demonstrations per task and the same environment budget. It tells us how much any shaping helps. On RoboCasa, the original paper reports that composite long-horizon tasks remain hard, "with success rates often below 12%", for imitation-based policies. Atomic-task success rose from 28.8% to 47.6% when 50 human demonstrations were augmented with 3,000 synthetic ones. These are behavior-cloning numbers and are reported as context, not as RL baselines.
The strongest existing method is static potential shaping with the same frozen critic, in the style of Robo-Dopamine. Its reported result is 95% real-robot success after 150 online rollouts, starting from 0% under sparse reward. That number comes from a physical setup we will not replicate exactly, and no reported result was found on Lacuna for static process-reward shaping on RoboCasa or Meta-World. We will therefore run it ourselves with identical critic, inner learner, demonstrations, environment steps, and an equal hyperparameter-search budget. Beating it means one of two outcomes. Either we reach its final success with clearly fewer environment steps on tasks where the critic is good. Or we reach higher final success on tasks where the critic is wrong (composite tasks and tasks with regressions), with no loss on the tasks where it is right.
The most dangerous competitor is outcome-supervised critic fine-tuning. Here the same adapter is trained by supervised learning as the robot gathers experience: the potential is regressed toward discounted success on the robot's own rollouts, with SARM-style "rewind" augmentation of failures (SARM used reversed failure clips to teach its model what regression looks like). It uses the same adapter, data, and compute.
We also run two other baselines. A BiPaRS-style bi-level baseline learns a state-action weight on the frozen critic's reward without the potential constraint, which tests whether the constraint matters. Hindsight relabeling of goals is included as the standard sparse-reward fix.
Three ablations isolate our ingredients. Removing demonstration resets from the outer objective should reproduce the zero-gradient stall. Dropping the potential form should show whether reward hacking returns. Replacing the adapter with full critic fine-tuning tests whether capacity or the objective is responsible for any gain.
Why not just…?
Why not just use the pretrained critic as a fixed potential? On many tasks this will be very hard to beat; Robo-Dopamine's 0% to 95% result says as much. Our bet is narrower. Gains should appear where the critic is miscalibrated: VLAC's reported failures on fumbles and cyclic motions, and RoboCasa's composite tasks. We test this directly by injecting known critic errors, as described below. If static shaping matches us on those tasks, the proposal fails, and we will report that.
Why not just fine-tune the critic on the robot's own success and failure outcomes? This is the real threat, and it shaped the design. Once successes exist, supervised relabeling is cheaper and may be just as good. We make two predictions, both falsifiable by our head-to-head comparison at equal compute. First, before the robot succeeds from true start states, supervised relabeling has only failures and can only push values down uniformly. The meta-gradient, fed by demonstration-reset successes, instead targets the states that change the next policy update. Second, supervised relabeling corrects prediction error everywhere, while the bi-level objective corrects only error that misleads the policy, which should matter when adapter capacity is small. If relabeling matches us across the critic-corruption experiment, we will conclude the bi-level loop is unnecessary.
Why not just have a language model rewrite the reward, as in the video-critique bi-level method? Language-Model-Assisted Bi-Level Programming has Gemini critique rollouts and GPT-4o rewrite reward code. It beat Eureka, but only on kinematic imitation in Isaac Gym, each outer step needing a full RL run and API calls. Its target is continuous motion matching, not contact-rich sparse success. We include it only as a cost reference: our outer step costs one extra backward pass through a small adapter.
Why not just do behavior cloning or hindsight relabeling? The file's evidence argues against both as end points. In SARM, vanilla behavior cloning reached 0% and 8% on T-shirt folding where reward-aware weighting reached 67% and 83%. Mezghani et al. found that hindsight relabeling failed on long, irreversible navigation where reachability-based shaping succeeded. Both remain in our baselines, but we do not expect them to compete on composite tasks.
Proposed Experiments
The main study runs on RoboCasa (code) using its 50-human-demonstration protocol on a subset of atomic and composite tasks. It reports success rate against environment steps, final success, and steps to reach the static baseline's final success, over at least five seeds. A secondary study on Meta-World (code) uses sparse success in place of its shaped rewards, to test whether one meta-learned adapter shared across MT10 tasks transfers. For reference, the benchmark paper reports multi-headed SAC at 85% on MT10 with its dense reward. We also use robosuite for the lift and door tasks, as fast sanity checks.
The decisive check is a controlled critic-corruption experiment modeled on BiPaRS's adversarial-shaping tests. We deliberately bias the frozen critic in three ways: assigning high progress to hovering near an object without grasping, making progress monotone through a regression segment, and inverting progress on one sub-stage. We then compare static shaping, outcome-supervised fine-tuning, unconstrained BiPaRS weighting, and our method.
The proposal is supported only if three conditions hold. First, our method recovers clean-critic performance under corruption faster than outcome fine-tuning, in the regime before the robot first succeeds from true start states. Second, removing demonstration resets collapses it to static-shaping performance. Third, the unconstrained variant shows loitering behavior that the potential form does not. If outcome fine-tuning matches us on all three, we will report that the simpler method suffices and that the potential constraint plus demonstration resets are the transferable lessons.
References
- Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation
- A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots
- On Learning Intrinsic Rewards for Policy Gradient Methods
- Learning Intrinsic Rewards as a Bi-Level Optimization Problem
- Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping
- Enhanced Meta Reinforcement Learning via Demonstrations in Sparse Reward Environments
- RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots
- SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation
- Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos
- Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping