Reinforcement learning (RL) agents excel at mastering short-term tactics, but they frequently stumble when faced with long-horizon tasks and sparse rewards. To cross a complex environment, an agent cannot simply think in terms of low-level motor primitives (like "move left leg 5 degrees"); it must reason at a higher level of abstraction (like "walk to the door").
Hierarchical Reinforcement Learning (HRL) and the options framework, formalized by Precup et al., provide the mathematical scaffolding for this multi-tier reasoning. Historically, however, HRL has suffered from a critical bottleneck: the high-level subgoals, task decompositions, and skill boundaries have been manually engineered by human experts. This reliance on human-specified structures creates brittle agents that fail to generalize to novel environments, scale poorly to complex tasks, and require intensive manual tuning.
To achieve true autonomy, recent research has shifted toward autonomous subgoal and hierarchy discovery. Instead of relying on human maps, modern hierarchical agents independently discover macro-actions, partition state spaces, and infer structural dependencies directly from environmental interaction.
The Failure Modes of Manual Decomposition
When human designers hand-craft subgoals or intermediate reward functions, they inadvertently restrict the agent's flexibility. As noted by Seong and Shim, manually designed behaviors are labor-intensive and often fail to capture the mathematically optimal strategies required for dynamic environments. If the environment changes slightly, a hard-coded task hierarchy becomes a liability rather than an asset.
Furthermore, manual decomposition limits scalability. In complex, end-to-end learning tasks, requiring human-authored walkthroughs or pre-trained skill sets prevents agents from operating in domains where expert data is unavailable (Drappo et al.; Tomar et al.). To build general-purpose decision-making systems, agents must learn to identify and organize sub-tasks autonomously from raw experience.
Discovering Structure via State Space Partitioning and Causality
How can an agent discover a subgoal without a human telling it what to look for? One powerful intuition is that the environment's state space contains natural geometric boundaries and bottlenecks.
For instance, Zhu and Dong demonstrate that algorithms like K-Nearest Neighbors and genetic algorithms can automatically partition large state spaces into distinct, manageable subtasks based on environmental transitions. By identifying the boundaries between these partitions, the agent naturally uncovers optimal subgoals (such as doorways or narrow passages) that connect different regions of the environment.
Beyond pure geometry, an agent can discover subgoals by modeling the causal prerequisites of a task. In long-horizon settings, certain milestones must be achieved before others become accessible. Peng et al. introduce a causality-driven approach that maps out the logical dependencies between environment variables. By understanding which states act as necessary gateways to subsequent rewards, the agent constructs a high-quality hierarchy that bypasses the need for inefficient, random exploration.
[Raw State Space] ──> [Causal/Geometric Analysis] ──> [Discovered Bottlenecks] ──> [Autonomous Subgoals]
Continuous Latent Spaces and Variable-Length Macro-Actions
Rather than forcing the agent to discover discrete, rigid milestones, another facet of autonomous discovery focuses on learning continuous representations of behaviors.
Co-Reyes et al. propose SeCTAR, a framework that utilizes a trajectory autoencoder to map complex action sequences into a continuous latent space of skills. Instead of picking a specific hard-coded sub-task, a high-level planner selects a coordinate within this continuous embedding space, which a low-level policy then executes. This provides the agent with fluid, adaptive control over its behavioral primitives.
To further alleviate the burden of step-by-step credit assignment, Fried et al. introduce Generative Proto-Sequence (GPS). GPS allows agents to make decisions at the sequence level rather than the action level by generating variable-length "macro-behaviors" in a single step. This mechanism effectively shortens the effective planning horizon, allowing the agent to leap across long temporal gaps without losing coherence.
End-to-End Optimization and Self-Play
Can an agent learn a hierarchy purely from a sparse, global reward signal without any intermediate reward shaping? The Option-Critic architecture, developed by Bacon and Harb, proves that it is possible. By deriving policy gradient theorems specifically for options, they enable the simultaneous learning of option selection, execution, and termination. The agent organically abstracts its action sequences into reusable components because doing so maximizes the global reward, entirely eliminating manual heuristics.
When external rewards are completely absent, agents can use intrinsic motivation and competitive dynamics to carve out their own hierarchies. Sukhbaatar et al. leverage asymmetric self-play, pitting two internal minds (Alice and Bob) against each other. Alice challenges Bob to reach various states in the environment. Through this automated curriculum, the system discovers a robust, continuous goal embedding space that a high-level controller can later exploit to solve downstream tasks.
Embracing Language and Foundation Models as Scaffolding
A recent and rapidly growing frontier in hierarchy discovery replaces abstract mathematical vectors with the compositional structure of natural language.
Jiang demonstrates that using natural language as the communication layer between high-level and low-level policies allows agents to generalize to entirely unseen instructions. However, to avoid the trap of relying on human-provided instruction sets, researchers are now integrating Large Language Models (LLMs) to automate the decomposition process.
Yang et al. introduce a framework that leverages LLMs for high-level guidance during an initial imitation phase. By using the pre-trained world knowledge of an LLM, the system automatically infers subgoals and structures, significantly reducing the computational cost and manual tuning traditionally required to kickstart hierarchical learning.