In the quest to build agents that can solve complex, long-horizon tasks, researchers have long turned to Hierarchical Reinforcement Learning (HRL). The intuition is simple: just as a human breaks down "make coffee" into "grind beans" and "boil water," an agent should use a high-level policy to select "skills" or "subgoals" that a low-level policy then executes.
However, this hierarchy introduces a new problem: the paradox of choice. As an agent becomes more capable and acquires a larger library of skills, the decision-making space for the high-level policy explodes. If every possible coordinate in a room is a potential subgoal, or every learned behavior is a potential action, the agent can become overwhelmed by the sheer number of options. This "utility problem," as described by Niekum et al., suggests that the cost of selecting the right skill can eventually outweigh the benefits of having it.
The Curse of Choice in Subgoal Spaces
In many HRL frameworks, the high-level policy treats the entire state space as its action space. For a robot in a 3D environment, this means the "action" of the high-level policy is a coordinate. Kim et al. point out that without effective pruning, the agent spends its time wandering aimlessly, picking subgoals that are irrelevant to the task.
This inefficiency is compounded when the subgoal space is high-dimensional. Li et al. argue that using raw state spaces as subgoal targets is often untenable. Their method, LESSON, instead learns compact latent representations by focusing on "slow" features—elements of the environment that change gradually—to make exploration more manageable. Similarly, Singh et al. note that scaling HRL to extremely high-dimensional objectives remains a primary technical hurdle, often leading to non-stationarity where the high-level policy chases "moving targets" that the low-level policy cannot yet achieve.
Restricting the Horizon: Adjacency and Reachability
One way to tame this explosion is to ensure the high-level policy only picks subgoals that are actually reachable. Guo et al. introduced HRAC (Hierarchical Reinforcement learning with Adjacency Constraint), which restricts the high-level policy to selecting subgoals within a "$k$-step adjacent" region. By preventing the agent from attempting physically impossible or overly distant goals, the search space is significantly pruned, leading to much higher sample efficiency.
A similar logic applies to navigation. Albore et al. use Reachability Graph Learning (RGL) to build a sparse map of the environment. By exploiting "translation invariance"—the fact that a "move left" skill works the same way regardless of absolute position—they reduce the complex problem of global navigation into a series of local, reachable transitions.
Managing the Skill Library
When the hierarchy is built on discrete skills (or "options") rather than continuous subgoals, the problem shifts to branching factor inflation. Konidaris et al. observed that as a robot accumulates macro-actions through trial and error, the increased size of the action space can actually slow down learning on new tasks.
To combat this, researchers have proposed several filtering mechanisms:
- Interest Functions: Khetarpal et al. proposed the Interest-Option-Critic, which learns "interest functions" that limit where a skill can be initiated. This prevents the high-level policy from even considering a "open door" skill when the agent is nowhere near a door.
- Skill Priors: The SPiRL framework learns a prior over behaviors from offline data. Instead of exploring all skills uniformly, the agent is guided toward behaviors that are likely to be useful in its current state, effectively narrowing the search space in complex environments.
- Retrieval and Pruning: Chauhan et al. developed OI-HRL, which uses a retrieval mechanism to index only relevant subtasks from a large library based on environment cues. For long-term library management, Evans et al. suggest that agents must eventually incorporate pruning strategies to identify and remove redundant or low-utility skills that clutter the hierarchy.
Toward Autonomous Structure
The ultimate goal of this research is to move away from manually designed hierarchies. Early work like HI-MAT by Mehta et al. sought to extract structural knowledge from successful examples to automate hierarchy design. More recently, Konidaris et al. demonstrated how agents can bridge the gap between low-level sensorimotor data and high-level symbolic reasoning through a "skill-symbol loop."
By solving the scalability of these libraries, we enable agents to not just learn skills, but to organize them into a coherent, navigable structure that grows in power without becoming paralyzed by choice.
Key Foundational Work
Clustering via Dirichlet Process Mixture Models for Portable Skill Discovery (2011) — contributed 2 concept elements to this direction.
Autonomous Skill Acquisition on a Mobile Manipulator (2011) — contributed 2 concept elements to this direction.
Automatic discovery and transfer of MAXQ hierarchies (2008) — contributed 1 concept elements to this direction.