Long robot tasks, such as opening a drawer, reaching in, placing an object, and closing it again, tend to fail at a few specific moments. Those moments are the narrow passages: the gripper must fit through a gap, line up with a slot, or clear a shelf lip. The volume of motions that succeed there is tiny. Current systems handle these moments in one of two ways. Some have a human mark the stage boundaries and hand-craft rewards that "jump" when a boundary is crossed. Others use a two-level controller, where a slow planner proposes intermediate goals and a fast policy reaches them, but the planner spaces those goals without knowing where the gaps are. We propose that the robot find its own bottlenecks. It would do this from the reachability estimates its low-level policy already learns, and then use those bottlenecks as the stage boundaries that the hierarchy, the exploration schedule, and the training budget all organize around.
In two sentences: a goal-conditioned policy's learned "how many steps from here to there" function reveals narrow passages as places many shortest routes are forced through. Placing subgoals at those places, and concentrating practice and exploration control around them, should replace hand-designed stage rewards and make long, tight-clearance tasks learnable from a sparse "goal reached" signal alone.
Core Claim
Two terms first. A goal-conditioned policy is a controller told which state to reach. Its value function can be trained with only a binary reached/not-reached signal and read as a temporal distance $d(s,g)$, the expected number of steps from state $s$ to goal $g$.
Build a graph over states the robot has visited, with edges weighted by $d$. A narrow passage then shows up as a node with high betweenness relative to local density. Betweenness here means the fraction of shortest graph paths between random state pairs that pass through the node. We define the bottleneck score
$$b(v) = \frac{\Pr_{(s,g)}[\,v \in \text{shortest path}(s,g)\,]}{\rho(v)},$$
where $v$ is a graph node, $(s,g)$ are start–goal pairs drawn from the replay buffer, and $\rho(v)$ is the number of visited states within a small temporal radius of $v$. The ratio is high where routes are forced through little space, which is the operational meaning of a narrow passage.
The claim is that nodes with high $b(v)$ serve three purposes at once. They are good subgoals for the high level. They are good starting states for low-level practice. They are reliable stage-crossing detectors for switching exploration off. Together, these should match or beat hand-engineered stage rewards on multi-stage manipulation, and degrade far more gracefully than uniformly spaced subgoals as physical clearance shrinks.
Relation to Prior Work
The closest work is Policy Transfer for Hierarchical Goal-Conditioned Reinforcement Learning (PROMO). It freezes a pretrained low-level policy, learns a reachability model, and places sparse landmarks that trade off reachability against novelty, then runs shortest-path search over the landmark graph. It supplies our graph machinery. Its landmarks are chosen for coverage, not for constriction. Its low-level policy stays frozen, so it cannot sharpen its skill where the geometry tightens. It was also evaluated mainly on AntMaze and Reacher, without tight-clearance manipulation. We keep its graph but change where landmarks go and what happens at them.
State-Dependent Maximum Entropy Reinforcement Learning for Robot Long-Horizon Task Learning (SDME) motivates the exploration component. It shows that suppressing exploration right after a milestone and boosting it when stuck cuts samples on 6–7-stage Meta-world tasks to about a third of SAC. It detects milestones through a hand-built reward with jump discontinuities. We keep its entropy logic and replace the detector with a learned one: crossing a high-$b$ node.
Two papers show that hand-set boundaries are a known weakness. CHD: Coupled Hierarchical Diffusion for Long-Horizon Tasks couples subgoal and trajectory generation through soft guidance but fixes segment boundaries by hand. Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (SuSIE) generates visual subgoals with no knowledge of what its low-level policy can reach.
The narrow-passage planning literature constrains the design and also inspires it. Multilevel motion planning: A fiber bundle formulation and Section Patterns win large speedups by planning in a lower-dimensional space where passages look wider, but they require hand-designed projections and a known geometric model. Our temporal-distance graph plays the role of a learned coarse space, obtained from interaction rather than specified.
Simultaneous Multi-Robot Motion Planning with Projected Diffusion Models is a cautionary result. Soft gradient guidance got 0% on corridor swaps where hard projection got 100%. We take this to mean that narrow passages must be handled explicitly rather than smoothed over, which is what bottleneck placement does.
We are honest about scope. True measure-zero constraints, such as closed-chain bimanual grasps, are beyond reach for sampled reinforcement learning. We target clearances that are small but have positive volume, like insertions with tolerance. The sweep below measures how small they can get.
Research Program
The system has four parts, built on a contrastive goal-conditioned learner of the kind analyzed in Demystifying The Mechanisms Behind Emergent Exploration in Goal-Conditioned RL, trained only on the sparse reached-goal signal.
First, the temporal-distance critic is periodically distilled into a graph over a subsample of replay states. The bottleneck score $b(v)$ is estimated by sampling start–goal pairs and running shortest-path search.
Second, the high level plans over this graph as PROMO does. However, subgoals are forced onto the highest-$b$ nodes along the route, and subgoals are placed more densely where $b$ is high.
Third, the low-level policy is not frozen. A fixed fraction of its training episodes are reset to states just before high-$b$ nodes, with the node beyond as the goal. This concentrates practice on the moves that fail.
Fourth, exploration temperature follows SDME's regime logic. The "up-stage" regime is triggered when the agent passes a high-$b$ node and its distance to the task goal drops sharply, instead of by a reward jump.
There is a chicken-and-egg problem: a passage must have been crossed a few times before betweenness can see it. Before that, the graph's frontier supplies novelty goals as in PROMO. A central analysis question is how many crossings the score needs before it stabilizes. We will also check whether the discovered nodes coincide with the stage boundaries humans annotated for SDME's reward, which gives a direct test of whether the learned boundaries are meaningful.
Baselines
The naive reference point is flat SAC with the sparse goal-reached reward only. It shows how hard each task is without any structure. On Meta-world long-horizon tasks, SDME reports that SAC variants need 7.5–10 million samples and generalize at best 44% under random initialization. We expect sparse-reward SAC to be worse still.
The strongest hand-engineered competitor is SDME itself, run with its published jump-property reward on drawer-put-in and box-pick-out. It reports convergence in about 2.5 million samples and 86% success under randomized initial states (State-Dependent Maximum Entropy Reinforcement Learning for Robot Long-Horizon Task Learning). SDME is the right bar because it encodes exactly the stage knowledge our method must discover. Beating it means matching 86% within the same sample budget while using no stage reward. Even equal performance is a meaningful result, since the human reward design is removed.
The strongest reward-free hierarchical competitor is PROMO, which reports near-100% success and 3–4× sample-efficiency gains in most AntMaze and Reacher environments. No reported result was found on Lacuna for PROMO on manipulation or narrow-clearance tasks, so we will run it ourselves. The flat contrastive learner, without a hierarchy, is a further baseline.
All methods share the same environments, goal specification, sample budget, network sizes, and hyperparameter-search budget (an equal number of tuning trials per method), and each is reported over the same seeds.
Three ablations isolate the key ingredient, bottleneck placement. The first keeps everything except that landmarks are placed by PROMO's reachability-plus-novelty rule at matched landmark count. The second keeps bottleneck placement but freezes the low-level policy and removes targeted resets. The third replaces bottleneck-triggered entropy control with a fixed temperature.
Why not just…?
Why not just use PROMO with more, denser landmarks? If subgoals are close enough together, every passage gets covered by accident, and bottleneck detection would be unnecessary. This is the most serious threat. The matched-count ablation and the clearance sweep settle it. If uniformly dense landmarks match bottleneck placement at every clearance, our central claim fails and we will say so. Our prediction is that uniform density wastes most of its landmarks in open space, and that the gap widens as clearance shrinks, because the fraction of landmarks landing in the passage scales with the passage's volume.
Why not just keep SDME's hand-crafted stage rewards? For a single benchmark task, this works: 86% generalization is strong. The cost is that each new task needs a new reward with carefully placed jumps, and the paper itself identifies that design as the brittle part. The test is whether learned bottleneck triggers recover SDME's performance without the reward. We also measure how SDME degrades when its jump points are shifted slightly, as a stand-in for imperfect human design.
Why not just plan through the narrow part with a geometric planner and learn the rest? When an accurate geometric model exists, this is a real alternative. Multilevel planners solve 37-DoF pre-grasping in under 2 seconds where standard planners time out (Section Patterns), and video-seeded RRT reaches "nearly 100%" on a Tunnel task where standard planners get 0% (Multi-step manipulation task and motion planning guided by video demonstration). Our setting assumes no hand-built geometric model or projection, which is what those methods require. We will still include an oracle-geometry planner on the Meta-world clearance variants as an upper reference. If it dominates, our contribution narrows to model-free settings, and we will state that.
Why not just use a large end-to-end policy? SuSIE's comparison showed the 55-billion-parameter RT-2-X failing on precision grasps where a decoupled system succeeded. This question also sits outside our reward-free reinforcement-learning regime, so we do not treat it as a primary baseline.
Proposed Experiments
The main testbed is SDME's Meta-world drawer-put-in (7 stages) and box-pick-out (6 stages). We add a parametric clearance sweep in which the drawer opening and box aperture shrink in steps toward the gripper width. We report success rate under random initialization and samples-to-convergence. The decisive plot is success versus clearance for bottleneck placement, uniform placement, SDME, and flat learners. Our hypothesis predicts a curve that stays flat much longer for our method.
The second testbed is the robosuite Shelf and Cluttered-Shelf tasks, where recovery policies on top of a nominal controller reach 83% and 57% (RecoveryChaining). Our method has no nominal controller, so this is a stretch comparison. It tests whether targeted resets at learned bottlenecks provide similar robustness at the shelf opening.
AntMaze and Reacher-wall serve as a sanity check that we do not lose PROMO's performance where passages are wide.
Two diagnostic checks decide whether the mechanism is real rather than incidental. First, the overlap between high-$b$ nodes and the human stage boundaries in SDME's reward, measured as the distance in steps between each discovered node and the nearest annotated boundary. Second, per-segment failure rates, which should fall specifically at bottleneck segments rather than uniformly.
The idea is falsified if either of two things happens. Discovered bottlenecks may not align with where failures occur. Or uniform landmarks at matched count may match performance across the clearance sweep.
References
- Policy Transfer for Hierarchical Goal-Conditioned Reinforcement Learning
- State-Dependent Maximum Entropy Reinforcement Learning for Robot Long-Horizon Task Learning
- CHD: Coupled Hierarchical Diffusion for Long-Horizon Tasks
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models
- Multilevel motion planning: A fiber bundle formulation
- Section Patterns: Efficiently Solving Narrow Passage Problems in Multilevel Motion Planning
- Simultaneous Multi-Robot Motion Planning with Projected Diffusion Models
- Demystifying The Mechanisms Behind Emergent Exploration in Goal-Conditioned RL
- Multi-step manipulation task and motion planning guided by video demonstration
- RecoveryChaining: Learning Local Recovery Policies for Robust Manipulation