Value-Surprise Gating: Letting a Learned Critic Decide When a Robot Should Replan and Where It Should Explore

Many current robot policies are trained by imitation and then executed in "chunks": the policy predicts a short sequence of motor commands, and the robot carries out the whole sequence without looking again. Recent work shows that a separately trained value function can push such a policy past its demonstrations. The value function is a critic, a network that predicts how likely the robot is to eventually succeed from a given situation. It scores several candidate chunks proposed by a frozen imitation policy, and the robot executes the best ones. This leaves two problems. A chunk runs blind even after physics has pushed the robot off course, which is exactly what happens when a policy built in simulation meets slightly different friction, mass, or latency. And if the frozen policy never proposes a good chunk in some state, the critic has nothing to promote, so the robot cannot learn its way out.

We propose to use one signal for both problems. That signal is the drop in the critic's predicted value while a chunk is executing. If the critic expected success when the chunk began and now, a few steps in, expects failure, something unplanned has happened. The robot should stop the chunk and propose again. If fresh proposals from the imitation policy still score poorly in that state, the robot has found a hole in its demonstration data, and only there should it spend exploration on actions the imitation policy would not generate. The central claim is that this "value surprise" finds the states where an imitation prior breaks down under a physics shift or a coverage gap. Concentrating replanning and exploration on those states should give more robustness and faster self-improvement than replanning or exploring uniformly, at the same compute and the same exploration budget.

Where this sits in prior work

The direct starting point is Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning. It keeps a large vision-language-action policy frozen as a source of candidate chunks and trains a roughly 1B-parameter off-policy critic on both successes and failures. At each decision it executes a critic-weighted average of $N$ proposals. It reports LIBERO-10 rising from 93% to 99%, RoboTwin from 83.8% to 91.4%, and real stack-cups from 40% to 90%. Its authors name the limitation we target: if the frozen policy never generates a successful chunk, the loop cannot bootstrap. The method also does not monitor execution inside a chunk. We reuse its critic and training recipe unchanged. What we add is a decision rule for when to use that critic during execution and when to leave the imitation policy's proposals.

APC-RL supplies the motivating evidence that a prior-free actor is necessary when demonstrations are misaligned: it reaches 100% on PointMaze where prior-only methods reach 0–7%. However, APC-RL decides between the prior and a free reinforcement-learning actor at every step, through value-weighted (Boltzmann) selection. It has been tested only in low-dimensional simulated control. Our position is that an always-available free actor is wasteful and destabilizing next to a strong pretrained policy. Exploration should switch on only where the critic has evidence that the prior is failing.

The replanning side is motivated by discrepancy-triggered methods. ATLAS halts and replans when observed outcomes diverge from look-ahead predictions, and reports 63.0% on WebArena-Lite against 53.9% for Plan-and-Act. Its discrepancy, however, is measured in a symbolic map of web-page changes. Lin et al. argue more generally that open-loop plans fail under unmodeled disturbances. We move the discrepancy from predicted observations to predicted value. Value is a single number already calibrated by the self-improvement loop, and it ignores physical deviations that do not matter for the task.

The physics-gap work constrains the design. Reinforced Grounded Action Transformation corrects each single-step action so the simulator matches the target dynamics. That does not fit chunked policies directly, and it was tested only on altered MuJoCo simulators. RialTo builds digital twins of real scenes and robustifies a flat policy with injected disturbances. It reports 90% versus 10% for imitation on "Book on Shelf," but it has no runtime replanning. Our method is complementary to both: it is a deployment-time mechanism that needs no model of the target physics.

Core mechanism

Let $Q(s, c)$ be the critic's value for starting chunk $c$ in state $s$. Let $V(s)$ be a small state-value head, distilled from $Q$ over the frozen policy's proposals, that is cheap enough to evaluate at every control step. Rewards are sparse and zero before success. So if a chunk committed at state $s_0$ is going as expected, discounting implies the value at step $j$ of the chunk should be about $Q(s_0, c)/\gamma^{j}$. We define the surprise at step $j$ as

$$\delta_j = Q(s_0, c) - \gamma^{j}\, V(s_j),$$

where $\gamma$ is the discount factor and $s_j$ is the state reached after $j$ steps of the chunk. Execution aborts when $\delta_j > \tau$, where $\tau$ is a threshold set from the distribution of surprise on successful rollouts (for example its 95th percentile).

After an abort, the robot draws fresh proposals and scores them as in Q-planning. If the best fresh proposal also scores below $Q(s_0,c)$ by more than $\tau$, the state is labeled a support gap. Only at support-gap states does a small residual actor add a learned correction to the best proposal's actions. The residual actor is a lightweight network trained off-policy to increase $Q$, with its output norm bounded. All transitions enter the same replay buffer, and only the critic, value head, and residual actor are updated. The large policy stays frozen, which keeps the stability that made Q-planning work.

Research Program

The work has three parts.

The first part asks whether the signal is informative, before any control claims are made. We will log $\delta_j$ on rollouts of a Q-planning agent and measure how well it predicts eventual failure (area under the ROC curve), separately under nominal physics and under shifted physics. If value surprise does not predict failure under physics shift, the critic is not calibrated off-distribution and the project stops there. That outcome would itself be a useful negative result.

The second part tests gated replanning alone at deployment, without additional learning. A critic trained under nominal simulation is deployed under shifted friction, object mass, and actuation latency, following the surrogate-target protocol of RGAT.

The third part adds gated exploration and runs the full self-improvement loop under the shifted physics. The shifted environment plays the role of "real." We measure how many iterations are needed to recover performance, compared with improving under shift without gating.

A small physical study on a dual-arm platform is desirable but not promised. We would use the 20 real teleoperated demonstrations per task that RoboTwin provides on its COBOT Magic platform.

Baselines

The naive reference is the frozen imitation policy executing full chunks with no critic. This is the starting point Q-planning reports before improvement, for example 83.8% on RoboTwin.

The strongest existing method is Q-planning itself, Beyond Imitation, run with its published recipe. Its reported numbers are 99% on LIBERO-10 and 91.4% on RoboTwin after improvement. LIBERO-10 is nearly saturated, so the comparison that matters is under physics shift. We found no reported result on Lacuna for Q-planning or any of these methods on LIBERO or RoboTwin with shifted dynamics, so we will establish those numbers ourselves. We will also report the RoboTwin tightly synchronized tasks such as "Dual Shoes Place," where RoboTwin reports success under 15%.

Fairness rules apply across all methods. Every method uses the same frozen policy, the same demonstrations, the same number of self-improvement iterations and environment steps, and the same tuning budget, which we fix at an equal number of hyperparameter configurations. Most importantly, every method gets the same total budget of critic evaluations per episode. Without this, gating could win simply by spending more compute. Beating Q-planning means higher success under shift at equal critic-evaluation budget, with non-overlapping confidence intervals over seeds. On nominal tasks, the gated method must stay within noise of Q-planning.

The second strong competitor is an APC-RL-style always-on prior-free actor with value-based arbitration. We port it to the same frozen policy and critic so that the only difference is that exploration is always available rather than gated.

Three ablations isolate the key ingredient. First, random aborts at the same abort rate test whether the timing of aborts matters. Second, fixed-interval replanning at the same number of replans tests the same thing against a regular schedule. Third, ungated residual exploration at the same total residual magnitude tests whether placing exploration at support-gap states matters. A fourth variant replaces value surprise with observation-prediction error from a learned forward model, in the spirit of ATLAS. It tests whether predicting value is better than predicting state.

Why not just…?

Why not just execute shorter chunks, or replan at every step? This is the most serious threat. If replanning more often is enough, our gate adds nothing. The answer depends on the cost of scoring: each replan evaluates $N$ proposals with a 1B-parameter critic, and Q-planning notes that this cost grows linearly. Smoothness also matters, since chunking exists partly to keep behavior coherent (Diffusion Policy). Our experiments plot success against critic evaluations per episode for fixed-interval replanning at several intervals, including every step, alongside the gated method. If fixed-interval replanning matches the gated curve at every budget, we will report that the replanning half of the idea is unnecessary. The remaining claim would then rest on gated exploration.

Why not just use heavy domain randomization? Randomization is known to make policies conservative on contact-rich precision tasks, as discussed in Overcoming the Reality Gap in Robotic Control. RoboTwin's synchronized tasks remain under 15% despite large variation in assets. Randomization and gating are also not exclusive. We will include a randomized-training variant of Q-planning and test whether gating adds to it. If randomization alone closes the shifted-physics gap at nominal cost, the deployment-time result is moot, but the support-gap exploration result would still stand.

Why not just fine-tune the large policy on its own successes? Filtered fine-tuning discards failures. Q-planning reports it stalling at 55% and 30% on real stack-cups and insert-wallet, where the critic-based method reached 90% and 80%. Full reinforcement-learning fine-tuning of the multi-billion-parameter policy risks destabilizing its representations, which is the reason Q-planning freezes it. We include filtered fine-tuning as a reference, but do not expect it to be competitive.

Why not just keep a prior-free actor available everywhere, as APC-RL does? This alternative could work, and the always-on baseline above tests it directly. Our expectation is that ungated exploration next to a strong prior costs success on states the prior already handles. If the always-on variant matches gated exploration in both final success and nominal-task success, gating is not needed for exploration.

Proposed Experiments

The evaluation uses LIBERO-10 and LIBERO-Goal from LIBERO, and the RoboTwin bimanual suite. Each is run under nominal physics and under three shifted-physics conditions (friction, object mass, and added actuation latency) at two severities. Metrics are success rate, abort rate, critic evaluations per episode, and iterations needed to recover nominal success under shift. We also report failure-prediction AUROC of $\delta_j$.

Three checks are decisive.

The first check is whether value surprise predicts failure under shift better than chance and better than observation-prediction error. This settles whether the signal exists.

The second check is whether, at matched critic-evaluation budget, gated replanning beats random and fixed-interval replanning under shift. This settles whether the timing of replanning matters.

The third check concerns support-gap tasks. We construct LIBERO-Goal variants where demonstrations of one sub-behavior are removed, so the frozen policy has zero support, mirroring APC-RL's misaligned-prior PointMaze. On these variants, gated exploration should bootstrap where Q-planning cannot, and should do so with less nominal-task degradation than the always-on actor. If it bootstraps only as well as uniform exploration, the localization claim fails.