This paper presents a demonstration-free approach to Hierarchical Reinforcement Learning (HRL) that decomposes complex navigation and manipulation tasks into a sequence of reusable skills. By training skills independently using "success-state resets," the authors achieve significantly better sample efficiency and generalization compared to standard "flat" RL baselines without requiring any expert demonstrations.
TL;DR
The authors propose a method for composing complex robotic behaviors by training individual skills sequentially, using the successful end-states of one skill as the starting points for the next. This demonstration-free pipeline significantly outperforms flat reinforcement learning in both discrete navigation and continuous manipulation, showing remarkable zero-shot generalization to unseen environments.
What Problem Does This Paper Address?
Standard Reinforcement Learning (RL) often fails on "long-horizon" tasks—tasks that require a long sequence of precise decisions before receiving a reward. For example, a robot navigating through multiple rooms or picking up and stacking blocks might only receive a reward at the very end. This "sparse reward" problem makes exploration incredibly difficult; the agent is essentially looking for a needle in a haystack.
While Hierarchical Reinforcement Learning (HRL) addresses this by breaking tasks into smaller sub-tasks (skills), many existing HRL methods rely on expert demonstrations to teach the agent how to transition between these skills. However, high-quality demonstrations are expensive and difficult to collect. This paper explores how to build reliable skill pipelines using only engineered rewards and a structured training curriculum.
Methods
The core of the approach is Policy Sequencing. Instead of training a single agent to do everything, the authors decompose the task into a chain of skills ($\tau_1 \to \dots \to \tau_K$).
The key innovation is Algorithm 1: Sequential skill training with success-state resets.
- Train Skill A: The first skill is trained from the initial environment distribution.
- Collect Successes: Once Skill A is proficient, the authors run it and save the full simulator states at the moment of success.
- Train Skill B: The next skill in the chain is trained using these saved "success states" as its starting points (resets).
This ensures that the "handoff" between skills is robust because the downstream skill is explicitly trained to handle the exact states produced by its predecessor.
In the Navigation domain (MiniGrid), they use parameterised skills:
GoTo: Navigates to a target cell within a room.CrossDoor: Moves through a doorway to the next room.
In the Manipulation domain (Robosuite), they use:
Reach,Grasp, andLiftskills, later extending the chain with aPlaceskill for stacking.
Datasets
The researchers evaluated their methods in two simulated environments:
- MiniGrid FourRooms: A discrete grid-world where an agent must navigate through doorways to reach a goal. The evaluation used 20 training layouts and 10 previously unseen layouts to test generalization.
- Robosuite: A continuous physics simulation using a Panda robot arm. They focused on "Pick and Lift" and "Stacking" tasks.
Key Insights and Ideas
- Demonstration-Free Alignment: You don't need expert data to align skills. Training downstream policies on the terminal state distribution of upstream policies is sufficient to create a reliable chain.
- Zero-Shot Generalization via Parameterisation: By making skills parameterised (e.g., telling the
GoToskill which coordinate to head toward), the agent can solve entirely new room layouts without any additional training, simply by following a new high-level plan. - Reduced Variance: Flat RL agents often exhibit high variance—some seeds succeed while others fail completely. The hierarchical approach showed much more consistent performance across different random seeds.
- Modular Extension: The "Pick and Lift" chain was successfully extended to a "Stacking" task by simply freezing the existing skills and training one new
Placeskill. This demonstrates the power of modularity in robotics.
Key Results
The hierarchical approach consistently outperformed "flat" baselines (DQN for navigation, SAC for manipulation):
- Navigation: On unseen layouts, the HRL pipeline achieved a 55.5% success rate, whereas the flat DQN baseline managed only 2%.
- Manipulation Efficiency: The HRL chain reached its success criterion in approximately 195,000 training steps, compared to 292,000 steps for the flat SAC agent.
- Stacking Success: The extended stacking pipeline achieved a 71.6% success rate.
Limitations and Future Work
While successful, the authors note several areas for improvement:
- Manual Planning: Currently, the high-level sequence of skills is manually specified. Future work could involve integrating an automated high-level planner.
- Grasp Reliability: In the manipulation tasks, the
Graspskill was the most common point of failure, suggesting that more complex reward shaping or control constraints might be needed for contact-rich interactions. - Automatic Skill Discovery: The current skills are pre-defined. A more autonomous system would discover and expand its own skill set as it encounters new transitions in the environment.
Summary updated