To solve complex, long-horizon tasks, reinforcement learning (RL) agents cannot reason solely at the level of raw motor torques or discrete grid steps. Instead, they must operate hierarchically, chunking primitive actions into reusable macro-actions—commonly referred to as skills, options, or concepts.
Yet, a fundamental design constraint persists across modern hierarchical reinforcement learning (HRL) and unsupervised skill discovery frameworks: fixed skill cardinality. Before training begins, researchers must manually specify a fixed hyperparameter $K$ representing the exact number of skills the agent is allowed to learn. This architectural bottleneck forces a rigid capacity on an agent's behavioral repertoire, severely limiting its ability to adapt, generalize, and scale.
The Goldilocks Dilemma of Skill Repertoires
Why is a fixed hyperparameter for skill capacity so problematic? It forces a compromise between two equally destructive failure modes:
- Under-expressiveness (Too Few Skills): If $K$ is set too low, the agent lacks the behavioral granularity required to navigate complex environments. In robotic manipulation, for instance, a rigid codebook size might be completely insufficient for multifaceted maneuvers that require distinct phases like grasping, aligning, and inserting. As noted in InfoCon: Concept Discovery with Generative and Discriminative Informativeness, forcing a system into a static concept capacity prevents it from adapting to tasks of varying complexity.
- Optimization Redundancy (Too Many Skills): Conversely, if $K$ is set too high, the agent wastes representation capacity. Sub-policies become redundant, optimization slows down, and sample efficiency plummets. As Jun Wang et al. demonstrate in Crowdfunding Dynamics Tracking: A Reinforcement Learning Approach, determining the optimal number of sub-policies currently requires extensive manual tuning and domain-specific knowledge, leaving models brittle when applied to new environments.
This rigid boundary means that an agent cannot autonomously determine the optimal level of behavioral abstraction needed for a specific environment, a limitation originally highlighted by Christian Daniel et al. in Probabilistic inference for determining options in reinforcement learning.
How Static Capacity Limits Generalization and Scale
The consequences of fixed skill cardinality ripple across several core subfields of reinforcement learning, from unsupervised exploration to multi-task transfer.
Unsupervised Exploration
In unsupervised skill discovery, agents maximize intrinsic objectives (like mutual information or state coverage) to map out environments without external rewards. However, the framework introduced in Exploration by Learning Diverse Skills through Successor State Representations points out that a predefined hyperparameter prevents the agent from automatically scaling its behavioral repertoire to match the physical size or complexity of a given environment.
Policy Transfer and Meta-RL
When transferring learned behaviors to new domains, a static skill library becomes a major bottleneck. Kuan Fang et al. show in Discovering Generalizable Skills via Automated Generation of Diverse Tasks that a static number of skills prevents a system from expanding its repertoire when encountering new environmental niches. Similarly, the authors of Toward Robust Long Range Policy Transfer note that relying on a fixed number of primitives hinders efficient scaling across diverse target domains. To achieve true coverage, Allan Jabri et al. argue in Unsupervised Curricula for Visual Meta-Reinforcement Learning. that frameworks must transition away from fixed discrete sets toward growing or continuous parameterizations of the skill set.
Multi-Task Offline Learning
When learning from mixed-task datasets, a fixed library capacity cannot adapt to the underlying data distribution. Jinmin He et al. observe in Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning that manual predefinition lacks the flexibility to scale the library's capacity based on the actual diversity and complexity of the training data.
Real-World Impact: Robotics and Navigation
In physical or high-fidelity simulated environments, the limitations of fixed cardinality become concrete barriers to performance.
- Robotic Assembly: In contact-rich manipulation tasks, Jiankai Sun et al. demonstrate in ARCH: Hierarchical Hybrid Learning for Long-Horizon Contact-Rich Robotic Assembly that assuming a finite, predefined set of low-level skills restricts an agent when a task demands entirely new fundamental actions.
- Motion Discretization: For learning manipulation concepts from unlabeled video demonstrations, Pei Zhou et al. argue in AutoCGP: Closed-Loop Concept-Guided Policies from Unlabeled Demonstrations that a static number of categories inherently limits a robot's ability to adapt as it encounters novel task requirements.
- Autonomous Navigation: In mapless navigation across unknown environments, architectures like the one proposed by Hyunki Seong et al. in Skill Q-Network: Learning Adaptive Skill Ensemble for Mapless Navigation in Unknown Environments face scaling limitations because a pre-set number of latent skills cannot dynamically adjust based on the specific navigation challenges encountered.
Pathways to Dynamic Skill Allocation
To break this bottleneck, researchers are developing methods that allow skill libraries to expand, chain, or compress dynamically based on environmental data rather than human intervention.
Fixed Cardinality (Rigid Bottleneck)
[Data] ──> [ Fixed Hyperparameter K ] ──> Limited Repertoire
Dynamic Cardinality (Adaptive Growth)
[Data] ──> [ Chaining / Bottom-Up Trees / Grammars ] ──> Scalable Repertoire
Soft Regularization
Rather than enforcing a hard-coded hierarchy, some approaches loosen the relationship between global skills and local tasks. Maximilian Igl et al. introduced Multitask Soft Option Learning, which utilizes task-specific policies regularized by a shared global prior. This allows individual tasks to deviate slightly from shared skills, mitigating local optima traps even when bounded by a predefined hyperparameter.
Skill Chaining
Instead of predefining a flat pool of skills, agents can discover skills sequentially by working backward from a goal. Akhil Bagaria et al. developed Option Discovery using Deep Skill Chaining, an approach where the environment's geometry and the agent's trajectories dictate the creation of new skills, allowing the hierarchical structure to adapt to the actual complexity of the environment.
Non-Parametric and Grammar-Based Growth
The most direct solution to the fixed cardinality problem is to build architectures that grow from the bottom up. Arthur Aubret et al. proposed ELSIM: End-to-end learning of reusable skills through intrinsic motivation, which autonomously constructs a hierarchical "tree of skills" to escape static constraints and discover granular action repertoires.
Taking a data-driven approach to long-horizon tasks, Damion Harvey et al. developed Unsupervised Hierarchical Skill Discovery. By combining optimal transport-based trajectory segmentation with grammar-based sequence compression, their framework adaptively determines the complexity of a task hierarchy directly from the data itself, bypassing the need for a predefined atomic skill budget.
By moving from fixed hyperparameters to open-ended, dynamic allocation, the next generation of hierarchical agents can scale fluidly alongside the complexity of the worlds they inhabit.
Key Foundational Work
Probabilistic inference for determining options in reinforcement learning (2016) — contributed 1 concept elements to this direction.
Hierarchical Reinforcement Learning Based on Return-Weighted Density Estimation (2017) — contributed 1 concept elements to this direction.