Scaling the Action Vocabulary of Robotic Manipulation

34 concepts34 papers1 proposal

When we design autonomous agents to interact with the physical world, we face an immediate challenge: the real world presents an infinite continuum of possible movements. To make this tractable, roboticists traditionally rely on action primitives—discrete, pre-programmed atomic behaviors like Pick(x, y) or Push(x, y, θ). By treating these primitives as the vocabulary of an agent, high-level planners can treat complex tasks as simple sequencing problems.

However, this abstraction introduces a severe bottleneck. If an agent's vocabulary is limited to just a few basic words, it remains functionally illiterate when facing the complexity of real-world environments. A growing body of research focuses on breaking this bottleneck by scaling the diversity, expressiveness, and scale of action primitive libraries.

The Primitive Bottleneck

Imagine trying to clear a tightly packed dishwasher or prepare a meal using only two physical actions: a straight-line top-down grasp and a horizontal push. When objects are cluttered, oddly shaped, or dynamically complex, these basic primitives fail.

Early work by Abdeslam Boularias et al. in Learning to Manipulate Unknown Objects in Clutter by Reinforcement demonstrated that restricting a robot to simple pushing and grasping severely limits its ability to resolve dense clutter, highlighting the open challenge of incorporating more sophisticated maneuvers like prying. When an object is flat against a surface and structurally "ungraspable," a standard pick primitive fails entirely. To overcome this, Kairui Ding et al. in PreAfford: Universal Affordance-Based Pre-Grasping for Diverse Objects and Environments expanded the action space to include "pre-grasping" primitives—such as flipping, rotating, or sliding an object against an environmental edge—to transition it into a graspable state.

[Standard Action Space] -------> Limited to Prehensile Grasps ------> Fails on Flat/Cluttered Objects
[Scaled Action Space]   -------> Pre-Grasping (Flip, Roll, Slide) --> Unlocks Complex Dexterity

This bottleneck extends far beyond household clutter. In specialized domains like laboratory automation, Marta Skreta et al. note in Large language models for chemistry robotics that a narrow set of primitives prevents robots from executing complex, multi-step chemical procedures. To perform tasks like recrystallization, the action library must scale to include highly specific physical manipulations like filtration and precise fluid sensing. Similarly, in human-robot collaboration, Emmanuel Senft et al. argue in Situated Live Programming for Human-Robot Collaboration that manufacturing workflows remain out of reach for end-user programming unless frameworks expand beyond basic movements to include rich interaction primitives like inserting and fastening.

How Action Diversity Unlocks Perception

The limitation of a small action vocabulary is not merely execution-centric; it fundamentally starves the robot's perceptual systems. An agent learns about the physical properties of its environment—mass, friction, articulation, and geometry—by interacting with it. If its interactions are monotonous, its understanding of the world remains shallow.

In Object Learning Through Active Exploration, Serena Ivaldi et al. show that active exploration models evaluated on a restricted subset of primitives suffer from limited sensorimotor feedback. To build robust object representations, the action repertoire must scale to encompass varied physical manipulations. This insight is reinforced by Lerrel Pinto et al. in The Curious Robot: Learning Visual Representations via Physical Interactions, who demonstrated that while self-supervised robots can learn visual representations by poking and grasping, a narrow action set only captures a fraction of an object's functional properties. To learn sophisticated geometric and functional features, the interaction vocabulary must include complex actions like pouring or stacking.

Furthermore, when dealing with complex or non-standard materials, rigid primitives fail to capture underlying physical realities. For instance, Priya Sundaresan et al. demonstrate in Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding that a robot cannot manipulate items with complex material behaviors (such as foods requiring twirling or scooping) without expanding its motion primitive diversity to reactively fuse visual and haptic feedback.

This interaction between action diversity and perception manifests across several core domains:

Mechanisms for Scaling the Action Vocabulary

If scaling the action vocabulary is necessary, how do we achieve it without making the action space impossibly complex to learn or engineer? Recent research focuses on three primary mechanisms: continuous parameterization, foundation models, and semantic reverse-engineering.

Continuous and Elaborate Parameterization

Instead of treating primitives as rigid, isolated macros, frameworks are moving toward hybrid representations that blend discrete action choices with rich, continuous parameters. In A Hybrid Approach for Learning to Shift and Grasp with Elaborate Motion Primitives, Zohar Feldman et al. address rigid action discretization by pairing discrete primitive selection with continuous 6-DOF parameterization. This allows the robot to execute highly nuanced maneuvers, such as subtly shifting an object to dynamically create a better grasping opportunity.

Leveraging Foundation Models and Human Data

To bypass the need to hand-engineer hundreds of unique action programs, researchers are leveraging large-scale human data and language models to discover and align action spaces. The authors of Structured World Models from Human Videos address the limitations of simplified action parameterizations (like discrete grasp points) by learning morphology-invariant action spaces directly from internet-scale human videos, allowing robots to interact with diverse tools.

To connect these rich actions to semantic commands, Kechun Xu et al. introduce the $A^2$ method in Efficient Alignment of Unconditioned Action Prior for Language-conditioned Pick and Place in Clutter, which aligns unconditioned action foundation models with vision-language priors to support complex functional behaviors like pouring.

Quantifying the Capability Gap

How large must a truly general-purpose action vocabulary be? To answer this, Jiefu Ou et al. presented a thought experiment called WorldAPIs: The World Is Worth How Many APIs? A Thought Experiment. By using Large Language Models to reverse-engineer thousands of wikiHow tutorials into executable action definitions, they induced over 300 unique APIs required for basic household utility.

[Traditional Simulators] --------> ~2-5 Primitives (Pick, Place, Push)
                                      |
                                      v  (Massive Capability Gap)
                                      |
[Real-World Human Needs] -------> 300+ Unique APIs (Unscrew, Filter, Wipe, Scoop)

This highlights a massive "capability gap" between current simulation benchmarks and the real world. In response, frameworks like GENMANIP, developed by Ning Gao et al. in GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation, use LLMs to automatically generate thousands of diverse manipulation tasks, forcing simulation environments to expand into non-prehensile skills like pouring, wiping, and tool use.

The Planning and Composition Challenge

Expanding a robot's action vocabulary from 3 primitives to 300 introduces a fundamental algorithmic challenge: the curse of dimensionality. If a hierarchical planner needs to sequence a 10-step task, choosing from 3 actions yields $3^{10}$ possible sequences. Choosing from 300 actions yields an intractable $300^{10}$ combinations.

To prevent this combinatorial explosion, researchers decouple abstract task planning from low-level geometric execution using structured representations.

Hierarchical Scene Graphs

One approach is to bound the action space using the semantic and geometric structure of the environment. In Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs, Jonathan Tremblay et al. manage a broad library of motion primitives by representing the world through a two-level scene graph. A high-level symbolic graph handles abstract relationships, while a low-level geometric graph handles precise 3D tracking, enabling rapid planning over long horizons. Similarly, Hanxiao Jiang et al. develop an Action-Conditioned Scene Graph in RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation to map how specific, intricate primitives (like unscrewing) reveal hidden states in the environment.

Hypergraph State Spaces

Instead of searching a standard sequential graph, task and motion planning (TAMP) frameworks can be restructured entirely. James Motes addresses the challenge of incorporating diverse manipulation strategies in Hypergraph-based Multi-Robot Task and Motion Planning by introducing the Decomposable State Space Hypergraph (DaSH). By representing the planning space as a hypergraph, the framework can efficiently compose complex new interaction rules and coordinate diverse actions across multiple objects simultaneously, accelerating solution times by orders of magnitude.

Traditional Graph Search:
[State A] --(Action 1)--> [State B] --(Action 2)--> [State C]  (Explodes with 300+ actions)

Hypergraph Composition (DaSH):
[Sub-Space 1] \
[Sub-Space 2]  ===[Hyper-Edge: Complex Multi-Object Primitive]===> [Target State]
[Sub-Space 3] /

By structuring the planning space around how actions geometrically and symbolically transform the world, robots can handle an expansive vocabulary of physical interactions without becoming paralyzed by choice.

Key Foundational Work

Learning to Manipulate Unknown Objects in Clutter by Reinforcement (2015) — contributed 1 concept elements to this direction.
Object Learning Through Active Exploration (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_action_primitive_diversity_8ab058a9,
  title = {action primitive diversity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/action-primitive-diversity/rdr_8ab058a93c134948980428eecfe00535},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_action_primitive_diversity_8ab058a9_v1,
  title = {action primitive diversity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/action-primitive-diversity/rdr_8ab058a93c134948980428eecfe00535/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake