Robots that learn long tasks from a single success signal at the end (the "sparse reward" setting) usually cut the task into intermediate targets called subgoals. Researchers used to hand-pick these subgoals; newer methods discover them automatically, choosing targets the low-level controller can reliably reach. This proposal argues that reachability is only half of what makes a subgoal useful. A subgoal is also the moment the robot checks its own work. If that check happens while the gripper blocks the camera, a failed grasp looks like a successful one. The error then passes unnoticed into the next segment and compounds until the episode is lost.
The idea is to discover subgoals that are both reachable and verifiable. They should end in states where the robot's cameras can tell success from failure, so that a "reach it or fail" rule can fire at the right moments. The underlying observation is that manipulation tasks contain moments where the gripper blinds the camera and moments where it does not. A learned hierarchy should place its checkpoints in the second kind, and it should treat a short retreat to regain sight as progress, not regress.
Core Claim
Hierarchical methods that discover subgoals from interaction or offline data choose intermediate targets purely by physical feasibility. Strict Subgoal Execution terminates an episode when a subgoal is missed and raises the cost of failure-prone graph edges. Guide to Control regularizes subgoals toward a reachability prior learned from offline data. EISP grounds its proposals with a hindsight sampler and a value function. All three work on low-dimensional ground-truth state, where checking whether a subgoal was reached is free and exact.
From pixels, that check becomes an imperfect classifier, and its errors concentrate where the arm occludes the scene. We claim that a hierarchy scoring candidate subgoals by reachability times the probability that the success check at that subgoal is correct will beat the same hierarchy scoring by reachability alone. The gain should grow with task length and with occlusion severity.
The reasoning follows from a split of per-segment outcomes. Let $R(g)$ be the probability of physically reaching subgoal $g$, and $A(g)$ the probability that the visual checker judges arrival correctly. A failure that is detected can be retried or recovered. A failure that is missed, with probability roughly $(1-R(g))(1-A(g))$, propagates silently. Chained over $n$ segments, the chance that nothing silent slips through is approximately
$$P_{\text{no silent failure}} \approx \prod_{i=1}^{n} \bigl(1 - (1-R(g_i))(1-A(g_i))\bigr),$$
which is the per-step product rule of Instruction Agent, applied only to undetected errors. When $A$ is high, a reach-it-or-fail rule turns low reachability into cheap retries. When $A$ is low, no rule can help. Selecting subgoals by $R \cdot A$ targets exactly the term that current methods ignore.
Research Program
We would build a two-level policy on image observations. The high level proposes subgoals as latent embeddings of future multi-view observations, sampled from a generative model trained on offline demonstrations in the style of Guider's latent subgoal model with its KL-regularized reachability prior. Alongside the generator, we train two heads. The first is a reachability head, trained with the reach-it-or-fail outcomes and edge-cost updates of Strict Subgoal Execution. The second is a subgoal verifier: given the current camera images and a subgoal embedding, it predicts whether the subgoal has been achieved.
During simulated training only, privileged simulator state gives ground-truth subgoal achievement. That lets us measure the verifier's per-state accuracy and train an accuracy head $A(g)$, which predicts how trustworthy the verifier's judgment will be once the robot arrives at $g$. At test time, no privileged state is used. The planner ranks candidate subgoals by $R \cdot A$, terminates segments on verifier-judged failure, and replans. The low level is a goal-conditioned visuomotor policy trained with the sparse terminal reward plus a subgoal-achievement reward from the verifier.
Two consequences follow from this design, and we test both.
First, subgoals should naturally split grasps into "approach to a visible pre-grasp pose" and "close and lift to a visible post-grasp pose" rather than ending mid-contact. That is the terminal blind spot that ContactGuard and Active Vision Might Be All You Need describe for wrist cameras.
Second, because reward comes from reaching verifiable subgoals rather than from a monotone progress score, a retreat that restores line of sight is rewarded. This addresses the monotonicity failure that VLAC acknowledges, where backtracking receives negative progress.
Several prior lines of work set constraints on this design. The relabeling-versus-strict-stopping tension between HAL and Strict Subgoal Execution matters here, because hindsight relabeling of occluded states is exactly how silent failures get rewarded. We therefore relabel only transitions whose achieved state has high $A$. The "proprioception pitfall" reported by ContactGuard, where joint states let a small-data latent model take shortcuts, warns against feeding raw joint states into the verifier. We inject proprioception only as separate conditioning, as DECO does, and ablate it.
Baselines
The naive reference point is a flat goal-conditioned visuomotor policy trained on the same demonstrations and fine-tuned online with the sparse terminal reward only. It has no subgoals and no verifier. It shows how much any hierarchy buys.
The strongest competitor on the precise question is Strict Subgoal Execution ported to pixels. It uses our subgoal generator, reachability head, verifier-based reach-it-or-fail termination, and edge-cost updates, but ranks subgoals by $R$ alone. This is the ablation that isolates the key ingredient: every component is identical except the accuracy term. It runs with the same offline data, the same number of online episodes, the same network sizes and the same hyperparameter search budget. Beating it means higher task success at matched interaction budgets on long RLBench tasks, together with a lower measured silent-failure rate. The file reports SSE only on state-based MuJoCo tasks, with nearly 100% on AntDoubleKeyChest where HIRO and HIGL failed entirely. No reported result was found on Lacuna for SSE, or any subgoal-discovery hierarchy, on pixel-based RLBench.
A second strong competitor is the best dense-reward flat learner: the same flat policy trained online with a pretrained progress critic. We would use Robo-Dopamine, which reports going from 0% to 95% success within 150 online rollouts with potential-based shaping, and VLAC, which reports 30% to 90% in 200 episodes on the AgileX PiPER. Both are given the same camera streams and the same episode budget. Beating them requires higher success on tasks with occluded contact phases, not merely matching them on short tasks.
A third baseline is a demonstration-keypose hierarchy, ChainedDiffuser. It reports 80.9% on challenging continuous-interaction RLBench tasks under imitation learning. We would fine-tune it with our verifier and reach-it-or-fail termination at its keyposes. This asks whether heuristic keyframe subgoals are already visible enough.
An upper bound replaces the verifier with privileged simulator state. It tells us how much headroom verification quality leaves.
Why not just…?
Why not just add more cameras, or re-render virtual views, so nothing is occluded? This is the most serious threat. ActiveVLA reaches 91.8% on RLBench with virtual viewpoint selection and zoom, and RVT re-renders point clouds from virtual orthographic views. Against this, ActiveVLA's own ablation shows active viewpoint selection is worth only 2.4 points on RLBench, and it doubles inference latency to 0.53 seconds. Wrist-camera blind spots at contact are geometric and persist with added viewpoints, as ContactGuard documents. We settle the question empirically by sweeping camera configurations: front only, front plus wrist, all RLBench cameras, and all cameras with RVT-style re-rendering. If the gap between our method and the reachability-only ablation vanishes with full re-rendering, we report that verifiability-aware placement matters only under sparse sensing. That is still useful for low-cost real robots.
Why not just use a learned progress critic as dense reward and skip subgoals? Critics like VLAC and Robo-Dopamine do densify reward, but they emit a scalar and not a checkpoint. They therefore provide no point at which to stop and retry. They also inherit the monotonicity assumption that penalizes the retreat-to-see behavior recovery needs. StepEval further reports that occlusion and off-camera actions cause misjudgments by frame-based evaluators. The critic baseline tests this directly. If it matches us on the long, occlusion-heavy tasks, subgoal verification is unnecessary.
Why not just take the subgoals from demonstration keyposes and add a verifier? Keyposes are chosen for motion, not visibility, and are often placed at the moment of gripper closure, which is the most occluded instant. The ChainedDiffuser-plus-verifier baseline measures this. A useful diagnostic is to compare the $A$ values of keypose subgoals against our discovered ones. If keyposes already have high $A$, the discovery component adds little and we will say so.
Why not just use touch? DECO shows a 20-point absolute gain from tactile input on its hardest tasks, and touch would reduce the need for visual verification during contact. RLBench has no tactile sensing, so our comparison is confined to vision. We treat touch as complementary: a tactile channel would raise $A$ for contact subgoals within the same scoring rule.
Proposed Experiments
The main testbed is RLBench (paper). We use its multi-stage tasks with sparse +1 rewards, a small set of demonstrations per task and a fixed online interaction budget, and group tasks by horizon length and by how much of the contact phase each camera configuration occludes. Occlusion is measured from simulator ground truth. THE COLOSSEUM supplies camera-pose and distractor perturbations that create occlusions the policy never saw. The file notes 30–50% drops for existing models there, and 65.9% for ActiveVLA under imitation. The robosuite Cluttered-Shelf task, where RecoveryChaining reports 57% and explicitly flags occlusion as its limit, tests whether verifiable subgoals make recovery from clutter-induced errors more reliable.
The reported RLBench and COLOSSEUM figures come from imitation protocols, not sparse-reward online learning. We therefore rerun every baseline under our protocol rather than comparing to published numbers.
Metrics are task success versus online episodes, the silent-failure rate (segments the verifier accepts but privileged state marks as failed), the number of segments per episode, and per-decision latency.
Three checks would decide the question. First, per-subgoal $A$ should predict downstream episode failure beyond what $R$ predicts; if it does not, the premise is wrong. Second, the advantage over the reachability-only ablation should grow with horizon length and shrink as cameras are added, as the product expression predicts. Third, the gap to the privileged-verifier upper bound should be small. If it is large, the bottleneck is verifier quality rather than subgoal placement, and the next step would be better perception, not better planning.
References
- Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning
- Guide to Control: Offline Hierarchical Reinforcement Learning Using Subgoal Generation for Long-Horizon and Sparse-Reward Tasks
- Implicit Subgoal Planning with Variational Autoencoders for Long-Horizon Sparse Reward Robotic Tasks
- Instruction Agent: Enhancing Agent with Expert Demonstration
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation
- A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots
- Language as an Abstraction for Hierarchical Deep Reinforcement Learning
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation
- ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation
- ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation
- RVT: Robotic View Transformer for 3D Object Manipulation
- Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulation
- RLBench: The Robot Learning Benchmark & Learning Environment.
- THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
- RecoveryChaining: Learning Local Recovery Policies for Robust Manipulation