Standard reinforcement learning (RL) often fails when rewards are sparse and horizons are long. If an agent only receives a "win" signal after thousands of individual motor commands, the probability of stumbling upon the correct sequence is effectively zero.
Hierarchical Reinforcement Learning (HRL) solves this by breaking tasks into a two-tier system: a high-level "manager" that selects subgoals and a low-level "worker" that executes the primitive actions to reach them. However, for decades, these hierarchies were hand-crafted by researchers. The emerging frontier in the field is autonomous discovery: enabling agents to identify their own subgoals, temporal abstractions, and task structures directly from environmental interaction.
The Manual Bottleneck
Early foundational work, such as the MAXQ framework by Tom Dietterich and the H-DYNA architecture by Satinder Singh, proved that hierarchy could turn "impossible" problems into manageable ones. By ignoring irrelevant state variables—a process Dietterich formalized as state abstraction—agents could learn significantly faster.
The catch was that these hierarchies required human experts to define the subtasks. As noted in Automatic discovery and transfer of MAXQ hierarchies, this manual design is time-consuming and limits an agent's ability to adapt to novel environments where the "correct" hierarchy isn't obvious to a human.
Finding the "Doorways": Bottleneck Discovery
One of the most intuitive ways to discover subgoals autonomously is to look for "bottlenecks" in the environment. Imagine a building with two rooms connected by a single door. To get anywhere in the second room, you must pass through that door.
The Q-Cut algorithm, developed by Ishai Menache, Shie Mannor, and Nahum Shimkin, treats the agent's transition history as a graph. By applying Max-Flow/Min-Cut algorithms to this graph, the system identifies these bottleneck states and automatically labels them as subgoals. Once identified, the agent creates "macro-actions" to reach these bottlenecks, effectively shrinking the distance between the start and the ultimate goal.
Learning Temporal Abstractions
Beyond specific states, agents can learn "options"—extended sequences of actions that satisfy a termination condition. The Option-Critic Architecture by Doina Precup and colleagues provided a breakthrough by allowing agents to learn these options end-to-end. Instead of a human saying "walking is a sub-skill," the agent learns which sequences of actions are worth grouping together based solely on the reward signal.
Similarly, John D. Co-Reyes and Pieter Abbeel introduced SeCTAR, which uses a trajectory autoencoder to learn a continuous latent space of skills. This allows a high-level planner to navigate the environment by picking points in "skill space" rather than individual actions, enabling the agent to solve long-horizon tasks that baffle "flat" RL models.
Language and Sketches as Scaffolding
While fully autonomous discovery is the goal, some researchers use "weak" supervision to bridge the gap. Jacob Andreas and Sergey Levine introduced Policy Sketches, which provide the agent with a high-level sequence of subtasks (e.g., "get wood, then make fire") without specifying how to do them.
Taking this further, Yiding Jiang and Kevin Murphy developed HAL (Hierarchical Abstraction with Language). HAL uses natural language as the communication layer between the manager and the worker. By using language, the agent can leverage the compositional nature of human speech to generalize to entirely new instructions it has never seen before.
The Bootstrapping Dilemma
A recurring challenge in this field is what Mehran Asadi and Manfred Huber call the bootstrapping dilemma: to discover useful subgoals, an agent needs a decent policy to explore the environment, but to learn a decent policy in a complex environment, the agent needs subgoals.
Recent work attempts to break this cycle using:
- Intrinsic Motivation: Cédric Colas (CURIOUS) uses "Learning Progress" to encourage agents to set their own goals in tasks they are just beginning to master.
- Experience Replay: Jeewon Jeon (MASER) extracts subgoals directly from the agent's past successes stored in a replay buffer, ensuring the subgoals are "actionable" and relevant to the team's success.
- Subgoal Search: Minglong Li (Dec-SGTS) allows agents to communicate high-level "intentions" rather than actions, reducing the overhead of coordination in multi-agent systems.
Summary of Intuition
The shift from manual to autonomous hierarchy is a shift from prescriptive to descriptive AI. Instead of telling an agent how to think about a problem (MAXQ), we are building agents that can look at their own experiences, identify the "bottlenecks" (Q-Cut), group their actions into "options" (Option-Critic), and eventually describe their plans through "language" (HAL). The end goal is an agent that can enter a completely unknown environment and, through exploration, build its own map of how to decompose the world into solvable pieces.