Hierarchical reinforcement learning splits a robot's control into two layers. A high-level policy picks a subgoal or skill every so often, and a low-level policy turns that choice into motor commands. On long tasks with rewards that arrive only at the end, this split is often the difference between learning and not learning. It depends, however, on someone deciding what the subtasks are. Methods that discover subtasks automatically exist, but they are almost always tested in discrete game worlds. On continuous manipulation benchmarks nobody has measured how much worse a discovered decomposition is than a human one, or why. We propose to measure this "decomposition gap" on a standardized manipulation benchmark where the human decomposition comes for free.
That benchmark is RLBench. Its demonstrations are generated by a motion planner that follows hand-designed waypoints, so every trajectory already carries the experimenter's manual task decomposition. The protocol keeps the hierarchical learner, data, and compute fixed and changes only where the subtask boundaries come from: human waypoints, fixed-length chunks, or one of several discovery methods. This makes manual versus autonomous decomposition a measured axis of the benchmark, in the same way RoboHive made state-based versus vision-based observation a measured axis. Our central hypothesis is that the field scores discovered hierarchies the wrong way. Agreement with human boundaries, the metric discovery papers report, should predict downstream success poorly. Whether the low-level controller can actually reach the end of each discovered segment should predict it well.
Core Claim
Discovery methods are currently evaluated by how closely their segments match a reference annotation. HiSD, for example, which segments unlabelled video into skill tokens and builds a skill grammar over them, reports 33% mean intersection-over-union against ground-truth segments in Minecraft. That number is the yardstick for its comparison with other segmentation methods. We claim that on continuous manipulation this kind of agreement is a weak proxy for whether the hierarchy helps. The quantity that matters is segment reachability, meaning how often a trained low-level skill reliably carries the robot from a segment's start to its end. Two lines of evidence suggest this.
The first is DIPPER. It trains the high-level policy with preference optimization and penalizes subgoals that the low-level critic rates as unreachable. Its ablation without that penalty drops sharply, which indicates that feasibility, not semantic correctness, is what breaks hierarchies. The second is Whiteson et al., who found in robot soccer that manual decompositions encode human biases that can block better policies. If that holds here, human waypoints are not even the right target to agree with.
If the claim is right, it has a direct consequence for method design. A discovery method should place boundaries only where reachability is high. We test this with a minimal method arm: candidate boundaries proposed by novelty, kept only if a goal-conditioned low-level critic predicts they are reachable.
Research Program
The first component is the protocol. We select long-horizon RLBench tasks with several waypoints from the 100-task suite (RLBench, resource). We generate a fixed offline demonstration set per task and keep the waypoint indices as the manual decomposition. RLBench's Task/Variation/Episode structure supplies held-out object and color variations, which we use as the transfer split.
We start with low-dimensional state observations. RoboHive reports a large performance gap between policies on ground-truth state and policies on pixels, and we do not want that gap mixed into the decomposition gap. A second phase repeats the protocol on RGB-D input to see whether the two gaps compound.
The second component is a fixed downstream learner. It follows a SPiRL-style design: a skill encoder and decoder trained on the offline segments, a learned prior over skills, and a high-level policy that selects skills and is fine-tuned with sparse task reward. Each skill runs until its segment's learned end condition fires. Low-level actions use chunk-wise delta end-effector commands. Feng et al. found across more than 13,000 real rollouts that this action format bounds error accumulation and beats step-wise delta by about 10%, so action-space choice should not confound the comparison. Every decomposition source uses exactly this learner, with matched gradient steps and environment interactions.
The third component is the set of decomposition sources. The first is human waypoints. The second is fixed-length chunks. The third is novelty-based decision points from NBDI, which ends skills where a curiosity model's prediction error spikes. The fourth is ordered subgoals from DTW alignment of demonstrations, following Paul et al., which assumes a linear subgoal chain with a preset count. The fifth is optimal-transport action segmentation as used in HiSD, applied to proprioceptive features without the grammar step, since that paper itself notes Sequitur breaks under continuous noise. The sixth is a control with random boundaries matched in segment count. The seventh is reachability-filtered novelty, our method arm.
We deliberately exclude symbolic discovery methods: CDHRL, which learns causal graphs over discrete inventory variables, and hierarchies of reward machines, which induce nested automata from given propositions. Both need pre-defined symbols that RLBench does not expose. We also exclude ARM-FM, which has a language model write reward machines. Its authors report that on Meta-World manipulation it needed human corrections for hallucinated spatial constraints, so it would not be a fully autonomous arm. These exclusions constrain the design rather than motivate it.
The fourth component is the measurements. Each run produces three numbers. The first is downstream task success and sample efficiency on training variations and on held-out variations. The second is boundary F1 against human waypoints within a temporal tolerance. The third is segment reachability, meaning the fraction of segment executions in which the frozen low-level skill ends within a fixed distance of the segment's end state. Reachability is close to DIPPER's lower-Q diagnostic, but we measure it by rollout rather than by critic estimate. The scientific test is a rank correlation, across all task and source pairs, of downstream success against boundary F1 and against reachability.
Baselines
The naive reference is the random-boundary control. It has the same number of segments, the same learner, and the same data, with boundaries drawn uniformly. Any discovery method that does not clearly beat it is not finding structure; it is only benefiting from temporal chunking. Alongside it we run two flat references. The first is SAC with an intrinsic curiosity bonus. The second is a flat imitation policy with chunk-wise delta actions trained on the same demonstrations. Both get the same interaction budget.
The strongest existing discovery competitor is NBDI. On D4RL Franka Kitchen, NBDI completed an average of 3.67 subtasks against 3.0 for fixed-length SPiRL. On a 40×40 maze it reached 0.25 success against 0.09, and on block stacking it stacked 1.12 blocks against 0.67 (NBDI). No reported result was found on Lacuna for any discovery method on RLBench, and no RLBench baseline success numbers were found either. We therefore re-run NBDI ourselves on RLBench using its released code, with the novelty percentile tuned on a held-out task set under the same tuning budget as every other arm. We also run the full protocol on Franka Kitchen (resource). If our NBDI reproduction there falls well short of 3.67 subtasks, the learner implementation is suspect and the RLBench results will not be trusted.
The upper reference is human waypoints. DIPPER reports up to a 40% success improvement over SAGA, HAC, and PIPER with hand-specified subgoals. No absolute number was found on Lacuna, so it serves as evidence that a well-regularized hierarchy on hand-designed subgoals is strong, not as a target figure.
The ablation isolating our key ingredient is reachability-filtered novelty against plain NBDI, with the same candidate boundaries and the critic filter as the only difference. Beating NBDI means higher held-out-variation success at matched interactions across a majority of tasks, with confidence intervals over at least five seeds. Closing at least half of the gap between NBDI and human waypoints would count as a substantive result.
Why not just…?
Why not just use the RLBench waypoints and skip discovery? On RLBench this works, which is why the waypoints serve as the upper reference arm. The point of the study is to price that convenience. Waypoints exist only because someone wrote a planner script for every task, and James et al. themselves frame that hand engineering as what keeps the suite fixed and artificial. If waypoints dominate every discovery method by a wide margin even on held-out variations, we will report it plainly: autonomous discovery would then be unjustified on standard manipulation suites, and that is a useful negative result. If they do not dominate, Whiteson et al.'s bias argument gets its first continuous-manipulation test.
Why not just use flat imitation with action chunking? This is the most serious threat. Feng et al. show chunk-wise delta is robust, and RLBench supplies unlimited demonstrations. Paul et al. found that pure imitation drifts on long horizons. In sparse multi-stage tasks, flat RL completed zero Kitchen subtasks in NBDI's evaluation, and in A² vanilla SAC reached about 20% on block stacking against about 90% with structured subgoals. None of that evidence comes from a flat chunked imitation policy on RLBench, though. We therefore run that policy as a reference. If it matches the hierarchical arms on held-out variations, the decomposition gap is moot for this suite, and we will say so.
Why not just report boundary agreement, as HiSD does? Agreement is cheap to compute and needs no downstream training. Our hypothesis is that it is misleading. The correlation analysis tests this directly. If boundary F1 predicts downstream success as well as reachability does, the core claim fails, and the cheaper metric should be adopted.
Why not just use fixed-length skills? NBDI already showed variable-length boundaries beat fixed-length ones across three D4RL domains. We still include fixed chunks because those gains may shrink when the action space is already chunked.
Proposed Experiments
The first decisive check is the correlation test. Across roughly ten long-horizon RLBench tasks and the seven decomposition sources, we compare the Spearman correlation of downstream success with reachability against its correlation with boundary F1. The claim survives only if the reachability correlation is clearly higher and the difference holds under a bootstrap over tasks.
The second check is the method arm. Reachability-filtered novelty must beat NBDI on held-out variations, and its gain must come with higher measured reachability. If it wins without higher reachability, the mechanism is not what we think it is.
The third check is transfer. Hierarchies discovered on training variations are reused unchanged on held-out object and color variations and, in the vision phase, under lighting and distractor shifts borrowed from the ★-Gen taxonomy. That paper found generalist policies degrade most under semantic and behavioral shifts. We ask whether discovered decompositions degrade more or less than human ones. A result where human decompositions transfer and discovered ones collapse would be the clearest evidence that current discovery methods fit the specific scenes they were trained on.
References
- Unsupervised Hierarchical Skill Discovery
- Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
- Evolving Soccer Keepaway Players Through Task Decomposition
- RLBench: The Robot Learning Benchmark & Learning Environment.
- RoboHive: A Unified Framework for Robot Learning
- Demystifying Action Space Design for Robotic Manipulation Policies
- NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
- Learning from Trajectories via Subgoal Discovery
- Causality-driven Hierarchical Structure Discovery for Reinforcement Learning
- Hierarchies of Reward Machines
- ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning
- Abstract Demonstrations and Adaptive Exploration for Efficient and Stable Multi-step Sparse Reward Reinforcement Learning
- A Taxonomy for Evaluating Generalist Robot Manipulation Policies