Cross-Category Generalization in Robotic Manipulation

43 concepts40 papers1 proposal

Historically, robotic manipulation has relied on highly specialized, ad-hoc strategies tailored to specific objects and environments (Building a Bridge Between AI and Robotics). While an industrial robot excels when handling uniform inventory in a structured setting, modern unstructured environments—such as e-commerce fulfillment centers or households—present an immense variety of object shapes, materials, and configurations (DoraPicker: An Autonomous Picking System for General Objects). Traditional end-to-end policies trained on a specific set of items fail to generalize when faced with novel geometries or variations in object pose (ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills).

To overcome this brittleness, a major paradigm shift has emerged: instead of learning how to manipulate entire objects as monolithic entities, recent frameworks decompose manipulation into generalizable sub-problems. This is achieved by leveraging part-level grounding, affordance priors, and semantic feature distillation.

Decomposing Objects into Actionable Parts

Why does a human know how to open a microwave even if they have only ever seen a kitchen drawer? The answer lies in functional compositionality: both objects share a common physical component—a handle connected to a hinge or slide.

By shifting a robot's focus from the global geometry of an object to its local, functional parts, models can achieve cross-category generalization. For instance, Geng et al. introduced PartManip to learn manipulation policies directly from point cloud observations of specific object parts rather than whole categories. Similarly, the authors of AdaRPG demonstrate that focusing on individual articulated parts allows a robot to adaptively infer and interact with novel mechanical structures on the fly, bypassing the need to recognize the object as a whole (GAMMA: Generalizable Articulation Modeling and Manipulation for Articulated Objects).

This part-centric view works because it drastically reduces the dimensionality of the generalization challenge. While the space of all possible household appliances is infinitely varied, the space of functional parts (handles, lids, buttons) is highly constrained. Geng et al. formalized this in SAGE by bridging semantic parts (what a part is called) with actionable parts (how it physically moves). This mapping allows natural language instructions to translate directly into robust physical interactions across entirely unseen object categories.

Distilling Semantic Priors into 3D Actions

Identifying these functional parts on a completely novel object requires a deep understanding of object affordances—the latent properties that dictate how an object can be used (Building Affordance Relations for Robotic Agents - A Review). Rather than training these affordance models from scratch using scarce robotics datasets, modern approaches distill this knowledge from pre-trained vision-language foundation models.

The intuition here is straightforward: 2D foundation models already contain rich, implicit representations of how humans interact with tools and objects. The core challenge is translating these abstract 2D semantic features into precise 3D spatial actions.

Wang et al. address this in SparseDFF by distilling semantic features from large 2D vision models into a 3D point cloud using only a few camera views. This allows a high-degree-of-freedom robotic hand to generalize a manipulation skill from a single demonstration to entirely new object categories and poses. Similarly, Tatiya et al. developed MOSAIC, which uses pre-trained language models to ground multi-sensory (visual, auditory, haptic) interactive experiences, enabling zero-shot generalization to downstream categorization and fetching tasks. By leveraging these foundational semantic priors, robots can predict functional, human-like grasps rather than awkward, purely geometric ones (Towards Affordance-Aware Robotic Dexterous Grasping with Human-like Priors).

Unifying and Normalizing the Action Space

Even with accurate part detection and semantic priors, execution can still fail if the robot's optimization space is inconsistent across different scenes or grippers. In cluttered real-world environments, mapping visual inputs to 6-DoF grasp poses is notoriously sensitive to minor variations in object orientation and sensor noise (6-DoF Grasp Pose Evaluation and Optimization via Transfer Learning from NeRFs).

To make grasp optimization robust and efficient, recent work focuses on unifying the representation space itself. Chen et al. proposed a "Normalized Grasp Space" (NGS) in Region-aware Grasp Framework with Normalized Grasp Space for Efficient 6-DoF Grasping. By mapping disparate grasp-relevant regions into a standardized coordinate system, their framework allows a single model to generalize seamlessly across cluttered scenes and different gripper types without losing inference speed.

When the action space is normalized or conditioned on flexible representations—such as the text-driven masks proposed by Wang et al. in Vision-Based Grasping through Goal-Conditioned Masking—the robot no longer needs to relearn the physics of grasping for every new geometry (Modeling Grasp Motor Imagery Through Deep Conditional Generative Models). Instead, it optimizes trajectories within a stable, invariant space. This mathematical consistency allows strategies to generalize whether the robot is manipulating rigid tools (TANGO), rearranging complex scenes (Consor), or shaping highly deformable objects like ropes and cloths (Deep Reinforcement Learning Based on Local GNN for Goal-Conditioned Deformable Object Rearranging; Learning Generalizable Tool-use Skills through Trajectory Generation).

Key Foundational Work

Fast Grasp Planning Using Cord Geometry (2015) — contributed 1 concept elements to this direction.
Building a Bridge Between AI and Robotics (1985) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generalization_limitations_in_robotic_gr_9322da14,
  title = {generalization limitations in robotic grasping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalization-limitations-in-robotic-grasping/rdr_9322da142e9544cdb19c4fb761ddcca1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_generalization_limitations_in_robotic_gr_9322da14_v1,
  title = {generalization limitations in robotic grasping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalization-limitations-in-robotic-grasping/rdr_9322da142e9544cdb19c4fb761ddcca1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake