Hierarchical Information Asymmetry for Reusable Skill Learning

16 concepts9 papers1 proposal

In hierarchical reinforcement learning (HRL), a high-level "manager" policy typically selects from a library of low-level "worker" policies, or skills, to solve complex tasks. A recurring failure mode in these systems is hierarchical collapse: if the low-level skills have access to the global task goal, they often over-specialize, becoming inseparable from the task-specific logic of the manager.

To solve this, a growing body of research proposes hierarchical information asymmetry. By intentionally blinding low-level policies to the final goal, researchers force these skills to become general-purpose motor primitives that capture the underlying dynamics of the environment rather than the specific requirements of a single task.

The Problem of Goal Overfitting

When an agent learns to stack blocks or navigate a maze, it is tempting to give every part of the neural architecture full visibility of the environment. However, as noted by Wulfmeier and Rao in Data-efficient Hindsight Off-policy Option Learning, providing full task context to low-level sub-policies often prevents the hierarchy from developing distinct, reusable components. If a "grasping" skill knows the ultimate goal is to build a tower, it may learn a movement that only works for that specific tower configuration, failing when the robot is later asked to put the block in a bin.

Salter et al. argue in Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning that this specialization is the primary hurdle to sample efficiency. Humans do not relearn how to reach or grasp for every new kitchen they enter; we reuse a stable library of motor skills. Information asymmetry is the architectural constraint that mimics this human capability.

Mechanism: The "Need to Know" Basis

The core idea of information asymmetry is to restrict the input space of the low-level policy. While the high-level controller sees the "global" state (e.g., the coordinates of a target), the low-level skills see only "local" or proprioceptive data (e.g., joint angles and velocities).

  1. Goal-Blindness: In Multitask Soft Option Learning, Igl et al. describe a framework where sub-policies are kept goal-blind. This forces them to learn behaviors that are useful across many different objectives.
  2. Gating and Composition: Chang et al. demonstrate in MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies that a high-level gating mechanism can use goal information to blend these goal-blind primitives. Because the primitives only focus on internal state dynamics, they can be combined simultaneously to solve high-dimensional challenges like walking while carrying an object.
  3. Unconditional Skills: Taking this to the extreme, Yunis et al. propose in Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learning that skills should be "unconditional"—independent even of specific observations from demonstrations—to maximize their transferability to new environments.

Information Asymmetry as Regularization

Beyond just building hierarchies, information asymmetry serves as a powerful regularizer. In Information asymmetry in KL-regularized RL, Galashov et al. introduce a "default policy" that is intentionally information-starved. By regularizing the main agent to stay close to this default policy, the authors force the agent to adopt structured patterns that are applicable across multiple tasks.

This concept evolved into behavior priors. The authors further demonstrate in Behavior Priors for Efficient Reinforcement Learning that hiding goal locations from a behavioral reference model forces it to internalize task-agnostic motor skills. These priors act as a "starting point" for the agent, significantly accelerating learning in environments with sparse rewards.

Why It Works: Environmental Regularities

The intuition behind this research direction is that the environment has structural regularities that are independent of any specific goal. Gravity, friction, and the agent's own morphology remain constant whether the agent is trying to run north or south.

As Veeriah et al. explain in Discovery of Options via Meta-Learned Subgoals, restricting sub-policies from accessing global goals forces the discovery of "environment-centric" behaviors. These behaviors capture how to interact with the world effectively. When the high-level policy eventually receives a new goal, it doesn't need to teach the agent how to move; it simply needs to learn which of the pre-existing, robust environment-centric skills to trigger.

This "division of labor," as Wulfmeier et al. describe in Compositional Transfer in Hierarchical Reinforcement Learning, is what allowed robots to learn complex manipulation tasks like block stacking on physical hardware in a fraction of the usual time. By ensuring skills remain goal-agnostic, the system ensures they remain useful forever.

Key Foundational Work

Compositional Transfer in Hierarchical Reinforcement Learning (2020) — contributed 2 concept elements to this direction.
Multitask Soft Option Learning (2020) — contributed 2 concept elements to this direction.
Information asymmetry in KL-regularized RL (2019) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_goal_agnostic_skill_learning_3db253e4,
  title = {goal-agnostic skill learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/goal-agnostic-skill-learning/rdr_3db253e49a4247e79109905ee127059a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_goal_agnostic_skill_learning_3db253e4_v1,
  title = {goal-agnostic skill learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/goal-agnostic-skill-learning/rdr_3db253e49a4247e79109905ee127059a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake