Preventing Skill Collapse in Hierarchical Reinforcement Learning

This direction is split into other directions: option collapse. Its last version stays readable and citable here.

Version 1 of 125 papers29 conceptsfrozen 2026-10-04

Hierarchical Reinforcement Learning (HRL) promises to solve complex, long-horizon tasks by breaking them into a library of reusable "options" or sub-policies. In theory, a high-level manager should be able to pick from a diverse toolkit of skills—like "walk," "climb," or "turn"—to navigate a difficult environment.

In practice, HRL often suffers from a phenomenon known as skill collapse or hierarchical collapse. Instead of discovering a diverse set of specialized behaviors, the sub-policies frequently converge to the same "average" behavior or a single redundant strategy. When this happens, the hierarchy effectively disappears, leaving the agent with a standard flat policy that lacks modularity and fails to generalize.

The Gravity of the Local Optimum

The primary reason hierarchies collapse is that reinforcement learning is inherently greedy. If a specific sequence of actions yields a reliable reward, every sub-policy in the hierarchy will feel the "gravitational pull" of that local optimum.

Without explicit pressure to differentiate, multiple options will specialize in the same high-reward mode. Ameet Deshpande and Balaraman Ravindran observed this in their analysis of Discovering hierarchies using Imitation Learning from hierarchy aware policies, noting that automated discovery often leads to "mode collapse" where sub-policies fail to partition the strategy space effectively.

When sub-policies are redundant, the high-level controller loses its "control authority." As Tuomas Haarnoja and Sergey Levine argue in Latent Space Policies for Hierarchical Reinforcement Learning, if the lower-level skills only cover a narrow range of behaviors, the higher-level policy cannot steer the agent into new states, even if it "wants" to.

Forcing Diversity through Regularization

To prevent this collapse, researchers introduce mechanisms that reward sub-policies for being different from one another. This is often framed as an information-theoretic or geometric constraint.

One approach is to use distance metrics to push sub-policies apart. Minsung Hyun and Nojun Kwak proposed using a Hellinger Distance regularizer to force options to learn mutually exclusive behaviors. By penalizing the similarity between the action distributions of different options, the agent is forced to "disentangle" its skills.

Another perspective, explored in Diversity-Enriched Option-Critic, uses intrinsic rewards based on mutual information. The goal is to maximize the "identifiability" of an option: given a sequence of actions, an observer should be able to easily tell which sub-policy was active. If all sub-policies look the same, this mutual information is zero.

Skill Discovery as Density Estimation

Instead of just penalizing similarity, some frameworks rethink how we find "modes" in the first place. Takayuki Osa and Masashi Sugiyama introduced Hierarchical Policy Search via Return-Weighted Density Estimation (HPSDE).

In HPSDE, the search for sub-strategies is treated as a density estimation problem. By identifying different clusters (modes) in the reward landscape, the algorithm can automatically determine how many distinct sub-policies are needed. This prevents the agent from collapsing into a single "average" policy by explicitly recognizing that a task might have multiple, equally valid ways to be solved.

Maintaining Modularity for Transfer

The ultimate goal of preventing collapse is to ensure that the learned skills are actually useful for future tasks. If an agent learns a diverse library of motion priors—as seen in the work of Chenhao Li and Marco Hutter on quadruped mobility—it can adapt to rugged terrain by combining and correcting those pre-trained skills rather than learning from scratch.

When the hierarchy is preserved, the agent achieves a "division of labor." Markus Wulfmeier and Abbas Abdolmaleki demonstrated in Regularized Hierarchical Policy Optimization (RHPO) that forcing this division allows robots to reuse motor skills across different manipulation tasks, significantly boosting data efficiency.

By treating options as distinct latent variables and applying robust off-policy updates—a method used in the HO2 algorithm—researchers can ensure that the hierarchy remains functional, avoiding the "flat policy" trap and enabling the emergence of complex, multi-stage behaviors.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_preventing_skill_collapse_in_hierarchica_2479b8e3,
  title = {Preventing Skill Collapse in Hierarchical Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preventing-skill-collapse-in-hierarchical-reinforcement-learning/rdr_2479b8e3f4404080b0073f0197f061e5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_preventing_skill_collapse_in_hierarchica_2479b8e3_v1,
  title = {Preventing Skill Collapse in Hierarchical Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preventing-skill-collapse-in-hierarchical-reinforcement-learning/rdr_2479b8e3f4404080b0073f0197f061e5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}