Hierarchical Reinforcement Learning (HRL) promises to solve complex, long-horizon tasks by breaking them into a library of reusable "options" or sub-policies. In theory, a high-level manager should be able to pick from a diverse toolkit of skills—like "walk," "climb," or "turn"—to navigate a difficult environment.
In practice, HRL often suffers from a phenomenon known as skill collapse or hierarchical collapse. Instead of discovering a diverse set of specialized behaviors, the sub-policies frequently converge to the same "average" behavior or a single redundant strategy. When this happens, the hierarchy effectively disappears, leaving the agent with a standard flat policy that lacks modularity and fails to generalize.
The Gravity of the Local Optimum
The primary reason hierarchies collapse is that reinforcement learning is inherently greedy. If a specific sequence of actions yields a reliable reward, every sub-policy in the hierarchy will feel the "gravitational pull" of that local optimum.
Without explicit pressure to differentiate, multiple options will specialize in the same high-reward mode. Ameet Deshpande and Balaraman Ravindran observed this in their analysis of Discovering hierarchies using Imitation Learning from hierarchy aware policies, noting that automated discovery often leads to "mode collapse" where sub-policies fail to partition the strategy space effectively.
When sub-policies are redundant, the high-level controller loses its "control authority." As Tuomas Haarnoja and Sergey Levine argue in Latent Space Policies for Hierarchical Reinforcement Learning, if the lower-level skills only cover a narrow range of behaviors, the higher-level policy cannot steer the agent into new states, even if it "wants" to.
Forcing Diversity through Regularization
To prevent this collapse, researchers introduce mechanisms that reward sub-policies for being different from one another. This is often framed as an information-theoretic or geometric constraint.
One approach is to use distance metrics to push sub-policies apart. Minsung Hyun and Nojun Kwak proposed using a Hellinger Distance regularizer to force options to learn mutually exclusive behaviors. By penalizing the similarity between the action distributions of different options, the agent is forced to "disentangle" its skills.
Another perspective, explored in Diversity-Enriched Option-Critic, uses intrinsic rewards based on mutual information. The goal is to maximize the "identifiability" of an option: given a sequence of actions, an observer should be able to easily tell which sub-policy was active. If all sub-policies look the same, this mutual information is zero.
Skill Discovery as Density Estimation
Instead of just penalizing similarity, some frameworks rethink how we find "modes" in the first place. Takayuki Osa and Masashi Sugiyama introduced Hierarchical Policy Search via Return-Weighted Density Estimation (HPSDE).
In HPSDE, the search for sub-strategies is treated as a density estimation problem. By identifying different clusters (modes) in the reward landscape, the algorithm can automatically determine how many distinct sub-policies are needed. This prevents the agent from collapsing into a single "average" policy by explicitly recognizing that a task might have multiple, equally valid ways to be solved.
Maintaining Modularity for Transfer
The ultimate goal of preventing collapse is to ensure that the learned skills are actually useful for future tasks. If an agent learns a diverse library of motion priors—as seen in the work of Chenhao Li and Marco Hutter on quadruped mobility—it can adapt to rugged terrain by combining and correcting those pre-trained skills rather than learning from scratch.
When the hierarchy is preserved, the agent achieves a "division of labor." Markus Wulfmeier and Abbas Abdolmaleki demonstrated in Regularized Hierarchical Policy Optimization (RHPO) that forcing this division allows robots to reuse motor skills across different manipulation tasks, significantly boosting data efficiency.
By treating options as distinct latent variables and applying robust off-policy updates—a method used in the HO2 algorithm—researchers can ensure that the hierarchy remains functional, avoiding the "flat policy" trap and enabling the emergence of complex, multi-stage behaviors.