Zero Shot Robotic Manipulation of Novel Object Geometries

This direction is split into other directions: zero-shot pose estimation. Its last version stays readable and citable here.

Version 1 of 1262 papers270 conceptsfrozen 2026-10-04

Traditional robotic systems operate under a "closed-world" assumption: they excel at manipulating objects they have seen thousands of times during training but fail when faced with a slightly different geometry. To move robots into unstructured human environments, researchers are developing methods for zero-shot generalization, enabling agents to interact with entirely novel objects without task-specific retraining or prior 3D models.

The Challenge of Geometric Variance

The primary hurdle in general-purpose robotics is the sheer diversity of object shapes. A policy trained to stack cubes often fails on a toy duck because it lacks an underlying understanding of stability and contact physics. As Alex X. Lee and colleagues demonstrated with the RGB-Stacking benchmark, there is a significant performance gap between mastering a fixed set of objects and generalizing to novel geometries Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes.

To bridge this gap, the field is shifting from learning specific object categories to learning geometric and functional invariants—features that remain consistent across different objects, such as corners, edges, and "graspable" surfaces.

Leveraging Geometric Foundation Models

One powerful intuition is that if a model understands 3D geometry in a general sense, it shouldn't need to be trained on a specific "mug" to know how to pick one up. Researchers are increasingly using "training-free" or "frozen" foundation models to extract these universal features.

For instance, the FreeZe framework Caraffa et al. achieves 6D pose estimation for unseen objects by fusing 3D geometric foundation models with 2D vision models FreeZe: Training-free zero-shot 6D pose estimation with geometric and vision foundation models. Similarly, Pos3R leverages the 3D-consistent features of the MASt3R model to estimate the pose of novel items using only a single RGB image, bypassing the need for massive synthetic datasets Pos3R: 6D Pose Estimation for Unseen Objects Made Easy. By treating geometry as a universal language, these systems can use a CAD model of a new part to interact with it immediately, a concept pioneered by methods like CorNet CorNet: Generic 3D Corners for 6D Pose Estimation of New Objects without Retraining.

Multimodal Binding for Physical Reasoning

Vision alone is often insufficient for novel objects because it cannot "see" physical properties like weight distribution, friction, or hardness. To solve this, researchers are "binding" touch to other senses.

UniTouch Fengyu Yang aligns diverse tactile sensor outputs into a shared multimodal embedding space with vision and language. This allows a robot to predict the stability of a grasp on a novel material without ever having felt it before Binding Touch to Everything: Learning Unified Multimodal Tactile Representations. Zexiang Guo and others have extended this to Large Language Models, enabling robots to "feel" and "see" simultaneously to infer properties like roughness or hardness in a zero-shot manner Robotic Perception with a Large Tactile-Vision-Language Model for Physical Property Inference.

Structural Priors as Interaction Guides

Instead of learning from scratch, robots can leverage the structural "priors" of the human hand or their own morphology. The intuition is that the way a hand is shaped dictates how it can interact with an object, regardless of the object's identity.

Iterative Refinement through Diffusion

Finally, for complex tasks like rearrangement or in-hand reorientation, the field is moving toward iterative generative models. Rather than predicting a single "correct" action, these models treat manipulation as a de-noising process.

RPDiff Anthony Simeonov uses relational pose diffusion on 3D point clouds to place objects in novel scenes, allowing the robot to "refine" the object's position until it fits the geometric constraints of the environment Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement. This iterative approach is also seen in DexDiffuser, which models hand-object interactions to eliminate "ghost states" (unphysical movements) when manipulating out-of-distribution objects DexDiffuser: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation.

By combining universal geometric features, multimodal physical grounding, and iterative planning, these researchers are moving robotics away from rigid, pre-programmed behaviors toward a flexible, "open-world" intelligence.

Papers in this version

Showing 60 of 262 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_zero_shot_robotic_manipulation_of_novel__1897994a,
  title = {Zero Shot Robotic Manipulation of Novel Object Geometries},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-robotic-manipulation-of-novel-object-geometries/rdr_1897994a71b84b929b8edca70856d03b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_zero_shot_robotic_manipulation_of_novel__1897994a_v1,
  title = {Zero Shot Robotic Manipulation of Novel Object Geometries},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-robotic-manipulation-of-novel-object-geometries/rdr_1897994a71b84b929b8edca70856d03b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}