Zero-Shot Skill Composition for Robotic Manipulation

47 concepts40 papers1 proposal

Imagine teaching a robot to prepare a meal. If the robot has to learn the exact sequence of movements for every single recipe from scratch, the training data required would be infinitely large. Humans avoid this combinatorial explosion by learning a discrete set of atomic skills—such as chopping, stirring, and pouring—and dynamically recombining them to execute entirely novel tasks.

In robotics, this capability is known as compositional generalization. As noted by Ayanna M. Howard et al. in ICRA 2022 Plenary and Keynote Speakers, assembling previously acquired skills into new sequences allows a robot to unlock a super-exponential number of behaviors without requiring additional training. Zero-shot skill composition is the mechanism that bridges the gap between isolated motor primitives and general-purpose, long-horizon robotic intelligence.

The Compositional Generalization Gap

Standard end-to-end robotic policies often suffer from a severe generalization gap. When a monolithic model is trained on specific pairs of actions and objects, it struggles to execute instructions that involve novel combinations of those familiar elements.

As Jiayuan Mao et al. explain in PDSketch: Integrated Domain Programming, Learning, and Planning, end-to-end systems fail at compositional generalization because they map high-level goals directly to low-level controls without intermediate abstractions. Similarly, Yilun Du et al. demonstrate in Learning Universal Policies via Text-Guided Video Generation that when a natural language command combines known concepts in an unseen configuration, monolithic networks cannot reliably extrapolate the physical actions required.

To overcome this limitation, the problem must be decoupled into two distinct phases:

  1. Skill Discovery: Isolating robust, reusable behavioral building blocks (primitives).
  2. Skill Chaining: Dynamically sequencing these primitives zero-shot to satisfy novel, complex objectives.

Discovering Reusable Primitives

Before a system can compose skills, it must possess a clean, distinct vocabulary of actions. Relying on high-level task descriptions alone makes it difficult to isolate where one behavior ends and another begins. To solve this, Zeren Chen et al. introduce a fine-grained approach in RH20T-P: A Primitive-Level Robotic Manipulation Dataset towards Composable Generalization Agents in Real-world Scenarios, demonstrating that explicitly defining and learning from primitive-level skills is foundational for downstream composition.

Rather than hand-coding these primitives, modern frameworks often discover them implicitly from unstructured data using information bottlenecks:

The Mechanics of Zero-Shot Chaining

Once a library of primitives exists, how do we stitch them together without retraining the underlying control policies? Researchers approach this through geometric, algebraic, and graph-based frameworks.

Trajectory Intersections and Graphs

One intuitive way to compose skills is to find where different behaviors naturally overlap in the physical world. In GTI: Learning to Generalize across Long-Horizon Tasks from Human Demonstrations, Ajay Mandlekar et al. show that an agent can link separate trajectories by identifying "intersections"—states where the environment configuration is identical across different demonstrations.

Taking this further, Max Argus et al. in Compositional Servoing by Recombining Demonstrations formalize imitation learning as a graph traversal problem. By partitioning long demonstrations into functional primitives, the robot can treat execution as finding a path through a searchable graph of partial demonstrations, effectively synthesizing entirely new task sequences on the fly.

Algebraic and Statistical Modularity

Alternatively, compositionality can be embedded directly into the mathematical structure of the control policy. Vitchyr Pong demonstrates in Composable Deep Reinforcement Learning for Robotic Manipulation that policies optimized via maximum-entropy Soft Q-Learning are inherently composable. Because the objective maximizes both reward and randomness, independently trained skills (such as "reaching" and "obstacle avoidance") can be linearly combined into a compound policy zero-shot, without further optimization.

From a statistical perspective, Qiangxing Tian et al. show in Independent Skill Transfer for Deep Reinforcement Learning that complex behaviors can be modeled as linear combinations of statistically independent primitive components extracted via Independent Component Analysis (ICA). This allows an agent to achieve fluid, multi-skill transitions by adjusting the weights of independent components rather than abruptly switching between discrete controllers.

Hierarchical Orchestration

To execute a complex instruction, a robot must map high-level linguistic concepts to these low-level physical primitives. This is achieved using hierarchical architectures that separate strategic planning from tactical execution.

┌─────────────────────────────────────────────────────────┐
│                 High-Level Reasoning                    │
│   (LLMs / VLMs / Program Parsers / Strategic Planning)  │
└────────────────────────────┬────────────────────────────┘
                             │
                             │ Parses instruction into
                             │ a sequence of primitives
                             ▼
┌─────────────────────────────────────────────────────────┐
│                 Low-Level Controllers                   │
│   (Atomic Skills / Reusable Primitives / Tactical)      │
└─────────────────────────────────────────────────────────┘

In Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks, Edwin Zhang et al. validate this separation by using a diffusion model to generate long-horizon strategic plans in a low-dimensional latent space, leaving the high-frequency tactical adjustments to low-level controllers.

To ground language into these plans, Renhao Wang et al. developed Programmatically Grounded, Compositionally Generalizable Robotic Manipulation. Their framework, PROGRAMPORT, parses natural language instructions into explicit, executable programs. By disentangling visual grounding from motor skills, the system can execute novel action-object pairings zero-shot because the language parser simply passes new arguments to the same underlying modular primitives.

Large Language Models (LLMs) can also perform this orchestration implicitly through context. In In-Context Learning Enables Robot Action Prediction in LLMs, Yida Yin et al. demonstrate that by converting visual scenes into text descriptions, off-the-shelf LLMs can leverage their in-context learning capabilities to sequence primitive skills into multi-step solutions without any weight updates.

The Frontier: Robust Transitions

While the theoretical foundation for skill composition is robust, real-world execution remains a major bottleneck. Evaluating this capability requires rigorous benchmarks that test models on out-of-distribution sequences.

The limits of current models are highlighted in ClevrSkills: Compositional Language And Visual Reasoning in Robotics, where Sanjay Haresh et al. show that state-of-the-art Vision-Language-Action (VLA) models almost completely fail to compose basic motor skills into multi-step tasks without explicit, task-specific training data. Similarly, Yangtao Chen introduces RoboHiMan: A Hierarchical Evaluation Paradigm for Compositional Generalization in Long-Horizon Manipulation, confirming that managing the physical transitions between skills under real-world perturbations is incredibly difficult for end-to-end architectures.

To ensure robust execution across these transitions, researchers are increasingly turning to structured neurosymbolic frameworks. For example, Andrew C Li et al. introduce Ground-Compose-Reinforce (GCR) in Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data, utilizing formal Reward Machines to logically enforce the boundaries and pre-conditions of composed skills. By wrapping learned primitives in logical rules, the agent can successfully navigate sparse-reward, multi-stage tasks zero-shot, ensuring that the composition is not just linguistically logical, but physically viable.

Key Foundational Work

Learning modular policies for robotics (2014) — contributed 1 concept elements to this direction.
Robust and Incremental Robot Learning by Imitation (2014) — contributed 1 concept elements to this direction.
A Holistic Approach to Compositional Semantics: A Connectionist Model and Robot Experiments (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_zero_shot_skill_composition_3b5856c3,
  title = {zero-shot skill composition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-skill-composition/rdr_3b5856c3cd004595835c27972dc43355},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_zero_shot_skill_composition_3b5856c3_v1,
  title = {zero-shot skill composition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-skill-composition/rdr_3b5856c3cd004595835c27972dc43355/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake