When we teach a robot to navigate a kitchen, prepare a meal, and clean the counter, we rarely expect it to learn the entire sequence as a single, continuous action. If it did, a slight change in the kitchen layout or a request to clean the counter before cooking would cause the system to fail completely.
Traditional reinforcement learning (RL) and imitation learning agents often suffer from this exact brittleness. Monolithic policies couple what the high-level objective is with how low-level motor controls are executed. As a result, they struggle with long-horizon tasks and fail to generalize to unseen scenarios.
The core intuition behind modular skill composition is to mimic human problem-solving: decompose complex, long-horizon tasks into a library of discrete, reusable sub-policies (primitives) and orchestrate them using an abstract, high-level planner. By treating skills as independent building blocks, an agent can achieve zero-shot generalization—solving entirely new tasks by simply rearranging its existing behavioral vocabulary without any additional training.
Decoupling Logic from Control
The primary mechanism that makes modular skill transfer work is the strict separation of concerns between high-level task logic and low-level physical execution.
Imagine an agent tasked with navigating a grid to collect items in a specific order. A monolithic policy must learn the entire path and the item sequence simultaneously. In contrast, a modular architecture splits this into two independent layers:
- The Low-Level Control Layer: Trained to achieve local, atomic goals (e.g., "move to the blue block" or "open the drawer"). It remains completely agnostic to the global task objective.
- The High-Level Planning Layer: Focuses entirely on abstract sequencing, routing, and logical constraints.
┌────────────────────────────────────────────────────────┐
│ High-Level Planner │
│ (Symbolic Logic, State Machines, Automata, LLMs) │
└───────────────────────────┬────────────────────────────┘
│ Sends Abstract Goal / Label
▼
┌────────────────────────────────────────────────────────┐
│ Low-Level Skill Library │
│ [ Skill A: Grasp ] [ Skill B: Move ] [ Skill C ] │
└───────────────────────────┬────────────────────────────┘
│ Executes Continuous Actions
▼
┌────────────────────────────────────────────────────────┐
│ Environment │
└────────────────────────────────────────────────────────┘
By separating these layers, the low-level skills become highly portable. As Illanes et al. demonstrate, decoupling symbolic goals from underlying control policies allows learned sub-skills to be reconfigured to satisfy entirely new objectives without retraining. Similarly, Zhang et al. show that training low-level policies to master transitions between adjacent abstract states creates reusable primitives that can be composed at inference time to solve novel, long-term goals.
This decoupling unlocks the ability to use formal logic for zero-shot execution. For example, Qiu et al. decouple goal-reaching skills from temporal logic objectives, enabling agents to follow intricate, multi-stage instructions (e.g., "Visit A, then B, while avoiding C") without ever seeing those specific combinations during training. Because the low-level controls are isolated, Araki et al. show that an agent can adapt to entirely new mission structures in just a few dozen steps by updating only its high-level meta-policy while keeping its underlying motor skills frozen.
Discovering and Representing the Skill Library
For this modular framework to work, we need a clean way to define, discover, and store these low-level primitives. If the skills are too specific, they won't generalize; if they are too broad, the high-level planning problem remains intractable.
Localized Training
One intuitive approach is to train the agent on localized, representative segments of a task rather than full compositions. In artistic stroke generation, Xie et al. show that mastering basic, localized brushwork primitives provides sufficient foundational knowledge to execute complex, full-scale compositions later on.
Unsupervised Skill Discovery
Instead of manually defining primitives, agents can discover them directly from unlabelled expert demonstrations. Kipf et al. introduce a differentiable segmentation module that breaks down continuous behaviors into a library of discrete "latent codes," allowing agents to generalize to task sequences longer than those seen during training. To make these primitives human-interpretable, Garg et al. utilize vector quantization to map language instructions to discrete skill abstractions, allowing a system to sequence behaviors zero-shot based on textual commands.
Decentralized Ensembles
Modularity can also emerge natively through competition. Goyal et al. propose an information-theoretic mechanism where independent sub-policies compete to act based on the state of the environment. Because there is no centralized master policy dictating their behavior, these primitives become self-nominating and highly transferable, allowing them to be easily plugged into entirely new task ensembles.
Orchestrating Primitives over Long Horizons
Once an agent possesses a robust library of skills, the final challenge is orchestration: how do we chain these primitives together to solve complex, long-horizon tasks? Researchers have converged on several powerful mechanisms to handle this high-level routing.
Finite State Machines and Automata
A natural way to sequence skills is through state transitions. Lin et al. integrate modular programmatic skills with a high-level finite state machine, enabling agents to handle repetitive or extended sequences by looping sub-routines across scaled environments. In a similar vein, the authors of Hierarchies of Reward Machines demonstrate that structuring rewards as modular, finite-state subroutines that can call one another prevents state-space explosion and allows complex behaviors to be assembled from simpler, known routines.
Symbolic Sketches and Programs
High-level symbolic guidance can serve as a flexible blueprint for execution. Andreas et al. introduce "policy sketches," which provide high-level hints about sub-task sequences without micromanaging low-level execution, allowing sub-policies to be shared and recombined across different tasks. For highly algorithmic domains, Pierrot shows that decomposing tasks into recursive, modular sub-routines allows agents to master recursive logic on minimal examples and generalize to problem instances of arbitrary size.
Predictive Dynamics Planning
Instead of relying on hardcoded logical rules, an agent can plan by predicting the outcomes of its skills. Agia et al. combine task-agnostic skill policies with forward dynamics models. By optimizing a plan that maximizes the predicted success probability of an entire skill sequence, the robot can generalize to novel, long-horizon manipulation tasks without ever needing to retrain the underlying skills themselves.
Why It Works: Combinatorial Generalization
The mathematical intuition behind why modular skill composition outperforms monolithic architectures comes down to combinatorial generalization.
Consider an environment where an agent needs to execute a sequence of $K$ steps, and there are $N$ possible types of atomic actions or sub-tasks.
- A monolithic policy views every unique sequence of length $K$ as a distinct, high-dimensional trajectory. To learn $M$ different tasks, it must explore and collect data for all $M$ trajectories independently.
- A modular policy focuses its exploration on mastering the $N$ individual primitives. Once those $N$ skills are learned, the agent can theoretically express and execute $N^K$ unique task combinations zero-shot, simply by changing the high-level sequence labels (den Hengst et al.).
By shifting the learning burden from "memorizing continuous end-to-end trajectories" to "routing discrete, pre-trained behaviors," modular skill composition fundamentally bypasses the curse of dimensionality. It transforms an intractable exploration problem in a massive continuous state space into a structured, high-level planning problem—unlocking a scalable path toward truly generalist agents.
Key Foundational Work
Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting (2012) — contributed 2 concept elements to this direction.
Transfer Learning in Sequential Decision Problems: A Hierarchical Bayesian Approach. (2012) — contributed 1 concept elements to this direction.
Evolution-Based Discovery of Hierarchical Behaviors (1996) — contributed 1 concept elements to this direction.