When we design autonomous agents to interact with the physical world, we face an immediate challenge: the real world presents an infinite continuum of possible movements. To make this tractable, roboticists traditionally rely on action primitives—discrete, pre-programmed atomic behaviors like Pick(x, y) or Push(x, y, θ). By treating these primitives as the vocabulary of an agent, high-level planners can treat complex tasks as simple sequencing problems.
However, this abstraction introduces a severe bottleneck. If an agent's vocabulary is limited to just a few basic words, it remains functionally illiterate when facing the complexity of real-world environments. A growing body of research focuses on breaking this bottleneck by scaling the diversity, expressiveness, and scale of action primitive libraries.
The Primitive Bottleneck
Imagine trying to clear a tightly packed dishwasher or prepare a meal using only two physical actions: a straight-line top-down grasp and a horizontal push. When objects are cluttered, oddly shaped, or dynamically complex, these basic primitives fail.
Early work by Abdeslam Boularias et al. in Learning to Manipulate Unknown Objects in Clutter by Reinforcement demonstrated that restricting a robot to simple pushing and grasping severely limits its ability to resolve dense clutter, highlighting the open challenge of incorporating more sophisticated maneuvers like prying. When an object is flat against a surface and structurally "ungraspable," a standard pick primitive fails entirely. To overcome this, Kairui Ding et al. in PreAfford: Universal Affordance-Based Pre-Grasping for Diverse Objects and Environments expanded the action space to include "pre-grasping" primitives—such as flipping, rotating, or sliding an object against an environmental edge—to transition it into a graspable state.
[Standard Action Space] -------> Limited to Prehensile Grasps ------> Fails on Flat/Cluttered Objects
[Scaled Action Space] -------> Pre-Grasping (Flip, Roll, Slide) --> Unlocks Complex Dexterity
This bottleneck extends far beyond household clutter. In specialized domains like laboratory automation, Marta Skreta et al. note in Large language models for chemistry robotics that a narrow set of primitives prevents robots from executing complex, multi-step chemical procedures. To perform tasks like recrystallization, the action library must scale to include highly specific physical manipulations like filtration and precise fluid sensing. Similarly, in human-robot collaboration, Emmanuel Senft et al. argue in Situated Live Programming for Human-Robot Collaboration that manufacturing workflows remain out of reach for end-user programming unless frameworks expand beyond basic movements to include rich interaction primitives like inserting and fastening.
How Action Diversity Unlocks Perception
The limitation of a small action vocabulary is not merely execution-centric; it fundamentally starves the robot's perceptual systems. An agent learns about the physical properties of its environment—mass, friction, articulation, and geometry—by interacting with it. If its interactions are monotonous, its understanding of the world remains shallow.
In Object Learning Through Active Exploration, Serena Ivaldi et al. show that active exploration models evaluated on a restricted subset of primitives suffer from limited sensorimotor feedback. To build robust object representations, the action repertoire must scale to encompass varied physical manipulations. This insight is reinforced by Lerrel Pinto et al. in The Curious Robot: Learning Visual Representations via Physical Interactions, who demonstrated that while self-supervised robots can learn visual representations by poking and grasping, a narrow action set only captures a fraction of an object's functional properties. To learn sophisticated geometric and functional features, the interaction vocabulary must include complex actions like pouring or stacking.
Furthermore, when dealing with complex or non-standard materials, rigid primitives fail to capture underlying physical realities. For instance, Priya Sundaresan et al. demonstrate in Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding that a robot cannot manipulate items with complex material behaviors (such as foods requiring twirling or scooping) without expanding its motion primitive diversity to reactively fuse visual and haptic feedback.
This interaction between action diversity and perception manifests across several core domains:
- Affordance Learning: In AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-Shot Interactions, Yian Wang et al. explain that restricting interaction types to basic primitives like pushing and pulling limits a robot's ability to infer hidden physical properties of articulated objects, such as joint limits or internal friction.
- Active Scene Exploration: When trying to identify hidden objects in dense piles, Zhenyu Wu et al. argue in Smart Explorer: Recognizing Objects in Dense Clutter via Interactive Exploration that relying on a single action type like pushing is highly inefficient. Resolving spatial ambiguities requires a diverse set of primitives, including grasping and suction.
- Interactive Navigation: The bottleneck even impacts mobility. Zhihai Bi et al. show in Interactive Navigation for Legged Manipulators with Learned Arm-Pushing Controller that mobile manipulators navigating through obstacles cannot rely solely on pushing; they must scale their skill sets to include pulling, lifting, and rotating to handle diverse environmental constraints.
Mechanisms for Scaling the Action Vocabulary
If scaling the action vocabulary is necessary, how do we achieve it without making the action space impossibly complex to learn or engineer? Recent research focuses on three primary mechanisms: continuous parameterization, foundation models, and semantic reverse-engineering.
Continuous and Elaborate Parameterization
Instead of treating primitives as rigid, isolated macros, frameworks are moving toward hybrid representations that blend discrete action choices with rich, continuous parameters. In A Hybrid Approach for Learning to Shift and Grasp with Elaborate Motion Primitives, Zohar Feldman et al. address rigid action discretization by pairing discrete primitive selection with continuous 6-DOF parameterization. This allows the robot to execute highly nuanced maneuvers, such as subtly shifting an object to dynamically create a better grasping opportunity.
Leveraging Foundation Models and Human Data
To bypass the need to hand-engineer hundreds of unique action programs, researchers are leveraging large-scale human data and language models to discover and align action spaces. The authors of Structured World Models from Human Videos address the limitations of simplified action parameterizations (like discrete grasp points) by learning morphology-invariant action spaces directly from internet-scale human videos, allowing robots to interact with diverse tools.
To connect these rich actions to semantic commands, Kechun Xu et al. introduce the $A^2$ method in Efficient Alignment of Unconditioned Action Prior for Language-conditioned Pick and Place in Clutter, which aligns unconditioned action foundation models with vision-language priors to support complex functional behaviors like pouring.
Quantifying the Capability Gap
How large must a truly general-purpose action vocabulary be? To answer this, Jiefu Ou et al. presented a thought experiment called WorldAPIs: The World Is Worth How Many APIs? A Thought Experiment. By using Large Language Models to reverse-engineer thousands of wikiHow tutorials into executable action definitions, they induced over 300 unique APIs required for basic household utility.
[Traditional Simulators] --------> ~2-5 Primitives (Pick, Place, Push)
|
v (Massive Capability Gap)
|
[Real-World Human Needs] -------> 300+ Unique APIs (Unscrew, Filter, Wipe, Scoop)
This highlights a massive "capability gap" between current simulation benchmarks and the real world. In response, frameworks like GENMANIP, developed by Ning Gao et al. in GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation, use LLMs to automatically generate thousands of diverse manipulation tasks, forcing simulation environments to expand into non-prehensile skills like pouring, wiping, and tool use.
The Planning and Composition Challenge
Expanding a robot's action vocabulary from 3 primitives to 300 introduces a fundamental algorithmic challenge: the curse of dimensionality. If a hierarchical planner needs to sequence a 10-step task, choosing from 3 actions yields $3^{10}$ possible sequences. Choosing from 300 actions yields an intractable $300^{10}$ combinations.
To prevent this combinatorial explosion, researchers decouple abstract task planning from low-level geometric execution using structured representations.
Hierarchical Scene Graphs
One approach is to bound the action space using the semantic and geometric structure of the environment. In Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs, Jonathan Tremblay et al. manage a broad library of motion primitives by representing the world through a two-level scene graph. A high-level symbolic graph handles abstract relationships, while a low-level geometric graph handles precise 3D tracking, enabling rapid planning over long horizons. Similarly, Hanxiao Jiang et al. develop an Action-Conditioned Scene Graph in RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation to map how specific, intricate primitives (like unscrewing) reveal hidden states in the environment.
Hypergraph State Spaces
Instead of searching a standard sequential graph, task and motion planning (TAMP) frameworks can be restructured entirely. James Motes addresses the challenge of incorporating diverse manipulation strategies in Hypergraph-based Multi-Robot Task and Motion Planning by introducing the Decomposable State Space Hypergraph (DaSH). By representing the planning space as a hypergraph, the framework can efficiently compose complex new interaction rules and coordinate diverse actions across multiple objects simultaneously, accelerating solution times by orders of magnitude.
Traditional Graph Search:
[State A] --(Action 1)--> [State B] --(Action 2)--> [State C] (Explodes with 300+ actions)
Hypergraph Composition (DaSH):
[Sub-Space 1] \
[Sub-Space 2] ===[Hyper-Edge: Complex Multi-Object Primitive]===> [Target State]
[Sub-Space 3] /
By structuring the planning space around how actions geometrically and symbolically transform the world, robots can handle an expansive vocabulary of physical interactions without becoming paralyzed by choice.
Key Foundational Work
Learning to Manipulate Unknown Objects in Clutter by Reinforcement (2015) — contributed 1 concept elements to this direction.
Object Learning Through Active Exploration (2014) — contributed 1 concept elements to this direction.