Robots that follow language instructions increasingly pair two kinds of models. A large vision-language model (VLM) reads the scene and the instruction and breaks the task into short language subgoals such as "pick up the red mug." A smaller, fast controller then turns each subgoal into motor commands. Current systems call the expensive model on a fixed schedule, or at every step. They also judge progress, which is needed because the only reward is a success-or-failure signal at the end of a long episode, by running another large model on every frame. This proposal asks the agent to learn, with a single policy, when each expensive call is worth paying for. At every moment the agent can keep going, pay a moderate price to have a critic check whether the current subgoal is done, or pay a large price to have the planner write a new subgoal. The signal driving this choice is the controller's own estimate of how likely it is to finish its current subgoal.
The payoff is a better trade-off between success and compute on long-horizon manipulation. There is also a side benefit. HAL shows that relabeling failed attempts in hindsight, by describing what the robot actually achieved as if that had been the goal, makes sparse-reward hierarchical learning work, but it needs a hand-coded oracle to write those descriptions. When critic calls are rationed, the VLM can play that oracle's role at an affordable cost.
Core Claim
Planning cost and verification cost should be decided jointly, not separately. The controller's subgoal-conditioned value, meaning its predicted chance of completing the current subgoal from the current observation, is the right quantity to base that decision on. It does three jobs at once:
- It flags subgoals the controller cannot reach, which calls for a replan.
- It flags likely subgoal completions, which calls for a critic check.
- In between, it justifies coasting.
A small learned gate that reads this value and chooses among three options is trained with an explicit compute penalty and a staged penalty schedule. We claim it will dominate, across the whole success-versus-compute curve, both of the following: replan-only gating in the style of Continue or Replan? (BCP), and always-on dense progress scoring in the style of VLAC.
The gate $g$ picks $a_t \in \{\text{coast}, \text{verify}, \text{replan}\}$ and maximizes
$$J(g) = \mathbb{E}\Big[R_{\text{task}} + \sum_t r^{\text{prog}}_t - \lambda\big(c_v N_{\text{verify}} + c_p N_{\text{replan}}\big)\Big].$$
Each term is defined as follows:
- $R_{\text{task}}$ is the sparse terminal success reward.
- $r^{\text{prog}}_t$ is the progress reward, issued only on steps where the critic was actually called.
- $N_{\text{verify}}$ and $N_{\text{replan}}$ count the calls made in an episode.
- $c_v$ and $c_p$ are their measured per-call costs in GPU-seconds.
- $\lambda$ sets the exchange rate between success and compute.
Rewards exist only where compute was spent. The gate is therefore trading reward density directly against cost, which neither BCP nor VLAC does.
Relation to Prior Work
Four lines of work motivate or supply this proposal, and each stops short of it in a different place.
Replan timing without subgoals. BCP trains a 16M-parameter continuation head with Group Relative Policy Optimization (GRPO), a policy-gradient method that scores each rollout against others sampled for the same input, to decide when a frozen VLA should re-plan. The replanning decision is rewarded with a Replanning-Efficiency Reward. BCP supplies our training recipe and is our strongest baseline. It cannot propose subgoals, cannot fix wrong action chunks, and has no notion of verification.
Cost-penalized mode choice in generation. InterVR penalizes multi-step planning inside a GRPO group when direct generation suffices. This is the closest analogue of our cost term, but it works on single-shot image generation rather than sequential control with environmental rewards.
Language subgoals with dense rewards but no cost control. HAL and VSC-RL use language subgoals to beat sparse rewards. VSC-RL reports a 7.8% drop when its VLM subgoals are removed, which motivates keeping the hierarchy. Neither prices its calls, and HAL depends on an oracle.
Critic cost and the risk of penalizing compute too early. VLAC motivates rationing the critic. Its pairwise-progress rewards lifted real-robot success from 30% to 90% in 200 episodes, but it scores every frame pair with an InternVL2-scale model. Zhu et al. constrain our design with a negative result: penalizing compute too early drives agents into degenerate short-cuts. For that reason $\lambda$ is ramped only after the gate reaches a success plateau.
Feasibility from the controller's value. DIPPER supplies the idea of using the lower level's value function to penalize infeasible subgoals. It applies this only to state vectors and uses it for training regularization. We reuse it on visual-language inputs as a runtime signal for when to replan.
Research Program
System. The high level is a frozen VLM prompted to emit one language subgoal from the current image and task instruction. The low level is a language-conditioned vision-language-action (VLA) policy. We add a value head that predicts subgoal completion. It is trained by regression on critic-labeled outcomes, so the only ground truth it uses is terminal episode success.
Critic role. The critic is a VLAC-style pairwise progress model. When called, it does three things:
- It scores progress since the last subgoal assignment.
- It declares the subgoal complete or not.
- On failure, it asks the VLM to caption what was actually achieved, which gives an oracle-free hindsight relabel for low-level training.
Gate. The gate is a small network whose inputs are the controller value, the value's recent change, the steps elapsed since the last call, and a compact visual embedding. It is trained with GRPO under the objective above, using the two-phase schedule: $\lambda = 0$ until the success rate plateaus, then a linear ramp.
Analyses. We will study three questions:
- Whether the controller value calibrates across the visual-language gap, measured by expected calibration error of predicted versus critic-judged completion.
- Whether gating concentrates critic calls near subgoal boundaries and on the non-monotonic stretches where VLAC is known to fail.
- Whether oracle-free relabels match HAL's oracle labels on MuJoCo-CLEVR, where the oracle is available as ground truth.
Baselines
Naive reference. The hierarchy replans at a fixed interval $K$ and runs the critic every $M$ steps. We sweep $K$ and $M$ to trace their own success-versus-compute curve. This baseline tests whether any learning is needed at all.
Strongest existing method. BCP is applied to the same hierarchy: its Bernoulli head decides replan-versus-continue, and the critic is never called except to give the terminal reward. On RoboTwin 2.0, BCP reports 93.94% average success against 89.88% for base LingBot-VLA, and an 11.08% gain on the 13 low-success tasks (BCP). We will reproduce BCP with the same base VLA, the same RL episode budget, and the same number of hyperparameter trials.
Beating BCP means two things: higher success at equal GPU-seconds per episode on the 13 low-success tasks, and a success-compute curve that lies above BCP's at every matched budget. On LIBERO, no reported result was found on Lacuna for BCP's success rate, so we will report our reproduction.
Dense critic. VLAC-style progress rewards are computed at every step. This gives the success ceiling that verification can reach and the compute ceiling our method tries to avoid.
Key ablations. Four ablations isolate what the proposal contributes:
- A two-way gate (coast/replan only) tests whether joint verification matters.
- A value-free gate tests whether the controller's value is the necessary input.
- A fixed-threshold rule on that value, with thresholds tuned on the same budget, tests whether learning the gate matters.
- HAL's oracle relabels on CLEVR give an upper bound for the relabeling component.
Why not just…?
Why not just replan every K steps? BCP shows that fixed windows cause timing misalignment and stale-observation failures. Its adaptive horizon raised hanging-mug success on AGIBOT G1 from 44% to 84%. Our sweep over $K$ reproduces this comparison inside the hierarchy rather than assuming it.
Why not just put BCP's head on top of a subgoal planner? This is the most serious competitor, and it is baseline two. Our argument is that a replan-only gate cannot buy reward density, so under sparse rewards it learns slowly. The two-way ablation settles the question. If it matches the three-way gate at every budget, then joint verification is not needed, and the contribution shrinks to "BCP with language subgoals."
Why not just threshold the controller's value? This is a real threat. If the value is well calibrated, a hand-set threshold may capture most of the gain. We treat this rule as an ablation with an equal tuning budget. We expect it to lose where calibration degrades: under the randomized RoboTwin setting, and on cyclic LIBERO-Long tasks where value is non-monotonic. If it does not lose there, we will report the simpler rule as the recommendation.
Why not just score every frame with a VLM critic? VLAC's own results show the reward benefit, but also the per-frame cost and its failure on non-monotonic tasks. The dense-critic baseline measures exactly how much success, if any, the gating gives up.
Proposed Experiments
Benchmarks. The main testbeds are RoboTwin 2.0, in both Clean and Randomized settings with emphasis on BCP's 13 low-success tasks, and LIBERO-Long for long-horizon language-conditioned manipulation (LIBERO). MuJoCo-CLEVR from HAL serves as the controlled testbed where oracle labels exist.
Metrics. We report:
- Success rate.
- Measured GPU-seconds, together with planner and critic calls per episode.
- Area under the success-versus-compute curve, traced by sweeping $\lambda$.
- Sample efficiency, measured as episodes needed to reach 80% of final success.
Decisive checks. The claim holds only if all of the following are true:
- The three-way gate's curve lies above both BCP's and the value-threshold rule's at matched compute.
- It reaches dense-critic success using less than a third of the critic calls. This target is a hypothesis, not a figure from the file.
- Delaying the $\lambda$ ramp prevents the early-collapse behavior Zhu et al. describe. We check this directly by comparing against a no-schedule run.
References
- Continue or Replan? Bernoulli-Continuation Policy Learning for Adaptive Horizon Execution
- A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots
- Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners
- Language as an Abstraction for Hierarchical Deep Reinforcement Learning
- VSC-RL: Advancing Autonomous Vision-Language Agents with Variational Subgoal-Conditioned Reinforcement Learning
- Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrieval
- Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning