In reinforcement learning (RL), we want agents to plan using high-level strategies rather than low-level, step-by-step actions. Instead of deciding how to move individual joint torques at every millisecond, an agent should ideally select a macro-action like "exit the room" or "fetch the keys." This concept is formalized through temporal abstraction, most notably via the options framework, where a high-level policy selects an "option" (a sub-policy), which then executes primitive actions until a termination condition is met.
When we attempt to train these hierarchies end-to-end, the architecture frequently breaks down. Instead of discovering a diverse library of modular, reusable skills, the system undergoes option collapse (or skill degeneracy). The mathematical structure of the hierarchy remains, but the actual behavior reverts to a standard, flat policy.
Understanding why this collapse happens—and how to prevent it—is a central focus of modern hierarchical reinforcement learning (HRL).
The Two Faces of Collapse
Option collapse typically manifests in one of two ways during end-to-end optimization:
1. Behavioral Degeneracy (The Monolithic Master)
In this failure mode, a single option expands to handle the entire task, or all available options converge to identical, mediocre "averages" of behavior.
As Klissarov et al. describe, this often stems from a "lazy" optimization loop. If one sub-policy happens to mature slightly faster early in training, the high-level policy will select it more frequently. This option receives more gradient updates, improving even further and suppressing all other options. This rich-get-richer feedback loop eliminates the advantages of modularity.
Without explicit pressure to differentiate, multiple options naturally gravitate toward the same local optimum of the reward function, failing to specialize in distinct modes of behavior, as noted by Smith et al. and Osa et al..
2. Temporal Collapse (Hyperactive Chatter)
The second failure mode destroys the temporal aspect of abstraction. Here, the high-level controller learns to terminate options almost immediately—often at every single timestep.
Zhang et al. identify this as "termination drift" or "chatter." Even if the high-level policy immediately re-selects the same option, the constant termination breaks the continuity of the skill.
Henaff et al. and Salter et al. observe that this occurs because a flat policy is inherently more flexible in the short term. To maximize immediate rewards, the high-level policy chooses to micro-manage the agent step-by-step rather than committing to an abstract, multi-step goal. This directly violates the core design principle of decoupled temporal roles popularized by architectures like FeUdal Networks (Vezhnevets et al.).
Why Does Collapse Happen?
Neural networks are fundamentally opportunistic. If an end-to-end gradient descent objective is given a flat reward function, it will find the path of least resistance to maximize that reward. It has no intrinsic preference for "elegant" or "human-interpretable" hierarchies.
Two main factors drive this opportunistic behavior toward collapse:
- The Availability Paradox: In standard HRL formulations, every option is considered valid and available in every possible state. Khetarpal points out that when skills are unconstrained by geography or context, there is no structural pressure for them to specialize. The network simply optimizes the most convenient option everywhere.
- Lack of Division of Labor: In single-task settings, a flat policy can often find a global optimum without needing modularity. Wulfmeier et al. argue that without explicit architectural or environmental pressure to enforce a division of labor, the hierarchical structure becomes redundant baggage that the gradient descent process eventually optimizes away.
Mechanisms for Preventing Collapse
To prevent the hierarchy from flattening, researchers introduce mechanisms that mathematically penalize collapse or structurally force specialization.
Restricting Scope via Interest Functions
If a single option can be run anywhere, it will try to do everything. We can prevent this by restricting where an option can be initiated.
Khetarpal et al. introduce Interest Functions to the option-critic framework. An interest function bounds an option's availability to specific regions of the state space. By forcing options to be context-specific, they naturally specialize in localized dynamics.
Taking this further, Abdulhai et al. introduce representation abstractions that restrict options not just in space, but to specific factors or features of the environment, ensuring that sub-policies remain independent and low-dimensional.
Latent Variable Inference and Hindsight Credit Assignment
Standard on-policy reinforcement learning only updates the option that was actively chosen during execution. This directly fuels the rich-get-richer problem.
To break this monopoly, Smith et al. treat options as latent variables within an Inferred Option Policy Gradient (IOPG) framework. Because the relationship between actions and options is treated probabilistically, a single experience can be used to update all options simultaneously via a differentiable inference step.
Wulfmeier et al. extend this concept to off-policy data using hindsight inference (HO2). By marginalizing over all possible option choices for a given trajectory, every sub-policy learns from every piece of data where its behavior matches the actions taken, ensuring balanced updates and preventing a single option from dominating.
Anchoring Termination to Environment Structure
To stop options from terminating prematurely (chatter), we can tie their termination functions to meaningful physical properties of the environment rather than letting them optimize freely against the flat reward.
Salter et al. introduce a predictability objective that incentivizes options to terminate specifically at environmental bottlenecks (like doorways or intersections). This ensures that options persist through open spaces and only hand control back to the manager when a structural transition occurs.
Optimization and Architectural Reframing
Other approaches modify the underlying optimization landscape to make collapse mathematically impossible or disadvantageous:
- Density Estimation: Osa et al. reframe hierarchical policy search as a return-weighted density estimation problem. By explicitly searching for distinct modes in the reward function, they force sub-policies to capture different strategies rather than collapsing into a single average policy.
- Parallel MDPs: Zhang et al. reformulate the options framework into two parallel, standard Markov Decision Processes (MDPs). This allows the use of stable, off-the-shelf algorithms like PPO to learn the termination function, eliminating termination drift.
- Soft Regularization: In multi-task settings, rigid skill sharing can cause optimization traps where updates are rejected if they temporarily hurt a single task. Igl et al. resolve this by using "soft" options regularized by a shared global prior, allowing individual tasks to deviate slightly and maintain distinct behaviors.
- Markovian Embeddings: Li et al. move away from the traditional Semi-Markov Decision Process (SMDP) formulation entirely, utilizing hidden temporal embeddings to stabilize the learning of multi-step skills.
- Meta-Learning Objectives: van Hoof frames option discovery as a meta-learning problem (FAMP), where sub-policies are optimized explicitly for how quickly a high-level agent can learn to compose them on new tasks, creating an inherent evolutionary pressure for a diverse, reusable skill library.
Key Foundational Work
Hierarchical Policy Search via Return-Weighted Density Estimation (2018) — contributed 1 concept elements to this direction.
An Inference-Based Policy Gradient Method for Learning Options (2018) — contributed 1 concept elements to this direction.
FeUdal Networks for Hierarchical Reinforcement Learning (2017) — contributed 1 concept elements to this direction.