Traditional robotic systems operate under a "closed-world" assumption: they excel at manipulating objects they have seen thousands of times during training but fail when faced with a slightly different geometry. To move robots into unstructured human environments, researchers are developing methods for zero-shot generalization, enabling agents to interact with entirely novel objects without task-specific retraining or prior 3D models.
The Challenge of Geometric Variance
The primary hurdle in general-purpose robotics is the sheer diversity of object shapes. A policy trained to stack cubes often fails on a toy duck because it lacks an underlying understanding of stability and contact physics. As Alex X. Lee and colleagues demonstrated with the RGB-Stacking benchmark, there is a significant performance gap between mastering a fixed set of objects and generalizing to novel geometries Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes.
To bridge this gap, the field is shifting from learning specific object categories to learning geometric and functional invariants—features that remain consistent across different objects, such as corners, edges, and "graspable" surfaces.
Leveraging Geometric Foundation Models
One powerful intuition is that if a model understands 3D geometry in a general sense, it shouldn't need to be trained on a specific "mug" to know how to pick one up. Researchers are increasingly using "training-free" or "frozen" foundation models to extract these universal features.
For instance, the FreeZe framework Caraffa et al. achieves 6D pose estimation for unseen objects by fusing 3D geometric foundation models with 2D vision models FreeZe: Training-free zero-shot 6D pose estimation with geometric and vision foundation models. Similarly, Pos3R leverages the 3D-consistent features of the MASt3R model to estimate the pose of novel items using only a single RGB image, bypassing the need for massive synthetic datasets Pos3R: 6D Pose Estimation for Unseen Objects Made Easy. By treating geometry as a universal language, these systems can use a CAD model of a new part to interact with it immediately, a concept pioneered by methods like CorNet CorNet: Generic 3D Corners for 6D Pose Estimation of New Objects without Retraining.
Multimodal Binding for Physical Reasoning
Vision alone is often insufficient for novel objects because it cannot "see" physical properties like weight distribution, friction, or hardness. To solve this, researchers are "binding" touch to other senses.
UniTouch Fengyu Yang aligns diverse tactile sensor outputs into a shared multimodal embedding space with vision and language. This allows a robot to predict the stability of a grasp on a novel material without ever having felt it before Binding Touch to Everything: Learning Unified Multimodal Tactile Representations. Zexiang Guo and others have extended this to Large Language Models, enabling robots to "feel" and "see" simultaneously to infer properties like roughness or hardness in a zero-shot manner Robotic Perception with a Large Tactile-Vision-Language Model for Physical Property Inference.
Structural Priors as Interaction Guides
Instead of learning from scratch, robots can leverage the structural "priors" of the human hand or their own morphology. The intuition is that the way a hand is shaped dictates how it can interact with an object, regardless of the object's identity.
- H-InDex Yanjie Ze et al. uses off-the-shelf 3D human hand pose estimators to provide a domain-specific structural prior, outperforming massive visual models on dexterous tasks H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulation.
- GeoMatch++ Yan Wei et al. models the relationship between an object's geometry and the robot's own physical structure (joints and links), allowing a policy to work on robot hands it has never seen before GeoMatch++: Morphology Conditioned Geometry Matching for Multi-Embodiment Grasping.
Iterative Refinement through Diffusion
Finally, for complex tasks like rearrangement or in-hand reorientation, the field is moving toward iterative generative models. Rather than predicting a single "correct" action, these models treat manipulation as a de-noising process.
RPDiff Anthony Simeonov uses relational pose diffusion on 3D point clouds to place objects in novel scenes, allowing the robot to "refine" the object's position until it fits the geometric constraints of the environment Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement. This iterative approach is also seen in DexDiffuser, which models hand-object interactions to eliminate "ghost states" (unphysical movements) when manipulating out-of-distribution objects DexDiffuser: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation.
By combining universal geometric features, multimodal physical grounding, and iterative planning, these researchers are moving robotics away from rigid, pre-programmed behaviors toward a flexible, "open-world" intelligence.