The Bottleneck of Fixed Skill Cardinality

16 concepts17 papers

To solve complex, long-horizon tasks, reinforcement learning (RL) agents cannot reason solely at the level of raw motor torques or discrete grid steps. Instead, they must operate hierarchically, chunking primitive actions into reusable macro-actions—commonly referred to as skills, options, or concepts.

Yet, a fundamental design constraint persists across modern hierarchical reinforcement learning (HRL) and unsupervised skill discovery frameworks: fixed skill cardinality. Before training begins, researchers must manually specify a fixed hyperparameter $K$ representing the exact number of skills the agent is allowed to learn. This architectural bottleneck forces a rigid capacity on an agent's behavioral repertoire, severely limiting its ability to adapt, generalize, and scale.


The Goldilocks Dilemma of Skill Repertoires

Why is a fixed hyperparameter for skill capacity so problematic? It forces a compromise between two equally destructive failure modes:

  1. Under-expressiveness (Too Few Skills): If $K$ is set too low, the agent lacks the behavioral granularity required to navigate complex environments. In robotic manipulation, for instance, a rigid codebook size might be completely insufficient for multifaceted maneuvers that require distinct phases like grasping, aligning, and inserting. As noted in InfoCon: Concept Discovery with Generative and Discriminative Informativeness, forcing a system into a static concept capacity prevents it from adapting to tasks of varying complexity.
  2. Optimization Redundancy (Too Many Skills): Conversely, if $K$ is set too high, the agent wastes representation capacity. Sub-policies become redundant, optimization slows down, and sample efficiency plummets. As Jun Wang et al. demonstrate in Crowdfunding Dynamics Tracking: A Reinforcement Learning Approach, determining the optimal number of sub-policies currently requires extensive manual tuning and domain-specific knowledge, leaving models brittle when applied to new environments.

This rigid boundary means that an agent cannot autonomously determine the optimal level of behavioral abstraction needed for a specific environment, a limitation originally highlighted by Christian Daniel et al. in Probabilistic inference for determining options in reinforcement learning.


How Static Capacity Limits Generalization and Scale

The consequences of fixed skill cardinality ripple across several core subfields of reinforcement learning, from unsupervised exploration to multi-task transfer.

Unsupervised Exploration

In unsupervised skill discovery, agents maximize intrinsic objectives (like mutual information or state coverage) to map out environments without external rewards. However, the framework introduced in Exploration by Learning Diverse Skills through Successor State Representations points out that a predefined hyperparameter prevents the agent from automatically scaling its behavioral repertoire to match the physical size or complexity of a given environment.

Policy Transfer and Meta-RL

When transferring learned behaviors to new domains, a static skill library becomes a major bottleneck. Kuan Fang et al. show in Discovering Generalizable Skills via Automated Generation of Diverse Tasks that a static number of skills prevents a system from expanding its repertoire when encountering new environmental niches. Similarly, the authors of Toward Robust Long Range Policy Transfer note that relying on a fixed number of primitives hinders efficient scaling across diverse target domains. To achieve true coverage, Allan Jabri et al. argue in Unsupervised Curricula for Visual Meta-Reinforcement Learning. that frameworks must transition away from fixed discrete sets toward growing or continuous parameterizations of the skill set.

Multi-Task Offline Learning

When learning from mixed-task datasets, a fixed library capacity cannot adapt to the underlying data distribution. Jinmin He et al. observe in Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning that manual predefinition lacks the flexibility to scale the library's capacity based on the actual diversity and complexity of the training data.


Real-World Impact: Robotics and Navigation

In physical or high-fidelity simulated environments, the limitations of fixed cardinality become concrete barriers to performance.


Pathways to Dynamic Skill Allocation

To break this bottleneck, researchers are developing methods that allow skill libraries to expand, chain, or compress dynamically based on environmental data rather than human intervention.

Fixed Cardinality (Rigid Bottleneck)
[Data] ──> [ Fixed Hyperparameter K ] ──> Limited Repertoire

Dynamic Cardinality (Adaptive Growth)
[Data] ──> [ Chaining / Bottom-Up Trees / Grammars ] ──> Scalable Repertoire

Soft Regularization

Rather than enforcing a hard-coded hierarchy, some approaches loosen the relationship between global skills and local tasks. Maximilian Igl et al. introduced Multitask Soft Option Learning, which utilizes task-specific policies regularized by a shared global prior. This allows individual tasks to deviate slightly from shared skills, mitigating local optima traps even when bounded by a predefined hyperparameter.

Skill Chaining

Instead of predefining a flat pool of skills, agents can discover skills sequentially by working backward from a goal. Akhil Bagaria et al. developed Option Discovery using Deep Skill Chaining, an approach where the environment's geometry and the agent's trajectories dictate the creation of new skills, allowing the hierarchical structure to adapt to the actual complexity of the environment.

Non-Parametric and Grammar-Based Growth

The most direct solution to the fixed cardinality problem is to build architectures that grow from the bottom up. Arthur Aubret et al. proposed ELSIM: End-to-end learning of reusable skills through intrinsic motivation, which autonomously constructs a hierarchical "tree of skills" to escape static constraints and discover granular action repertoires.

Taking a data-driven approach to long-horizon tasks, Damion Harvey et al. developed Unsupervised Hierarchical Skill Discovery. By combining optimal transport-based trajectory segmentation with grammar-based sequence compression, their framework adaptively determines the complexity of a task hierarchy directly from the data itself, bypassing the need for a predefined atomic skill budget.

By moving from fixed hyperparameters to open-ended, dynamic allocation, the next generation of hierarchical agents can scale fluidly alongside the complexity of the worlds they inhabit.

Key Foundational Work

Probabilistic inference for determining options in reinforcement learning (2016) — contributed 1 concept elements to this direction.
Hierarchical Reinforcement Learning Based on Return-Weighted Density Estimation (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fixed_skill_cardinality_474a9583,
  title = {fixed skill cardinality},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-skill-cardinality/rdr_474a9583db594453b0823677a40b989e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_fixed_skill_cardinality_474a9583_v1,
  title = {fixed skill cardinality},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-skill-cardinality/rdr_474a9583db594453b0823677a40b989e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake