When a human interacts with a novel object—such as an oddly shaped modern sculpture or an unfamiliar kitchen gadget—we do not need to retrain our brains or consult a mental CAD catalog to grasp it, rotate it, or predict how it will roll. We rely on an intuitive understanding of geometry, local curvature, and physical space.
In contrast, traditional computer vision and robotic manipulation systems have long been bound to category-specific templates. If a model is trained to estimate the pose of chairs, it typically fails when presented with a lamp.
A rapidly growing body of research bypasses this limitation through zero-shot geometric generalization. Instead of memorizing what specific object categories look like, these systems learn the underlying principles of 3D space, contact physics, and spatial boundaries. By decoupling geometry from semantic identity, these models successfully interact with, reconstruct, and reason about entirely unseen object categories out of the box.
The Primitive-to-Complex Transfer
Why can a model trained on simple shapes generalize to highly complex, irregular real-world objects? The core intuition lies in the concept of local linearity and curvature approximation: at a microscopic level, the contact patch between a robotic fingertip and a complex object resembles the surface of a simple geometric primitive, such as a cylinder or a sphere.
[Complex Real-World Object]
│
▼ (Zooming into local contact patches)
[Patch A: Convex Curvature] ──► Approximated as a Cylinder
[Patch B: Flat Surface] ──► Approximated as a Plane
[Patch C: Point Contact] ──► Approximated as a Sphere
By forcing a model to master contact mechanics on basic geometric primitives, researchers prevent the network from memorizing global object topologies. For instance, in developing a system for in-hand object rotation, Haozhi Qi et al. trained an adaptive controller exclusively on simple cylinders in simulation. Because the policy was forced to rely on local fingertip interactions and curvature properties rather than a global shape template, it achieved robust zero-shot generalization when deployed on highly irregular real-world objects like shuttlecocks and baseballs in their framework, HORA.
This principle applies broadly across manipulation and localization tasks:
- Motor Policies: Yifeng Jiang et al. demonstrated in DASH that motor policies trained solely on primitive geometric shapes can successfully execute complex manipulation tasks, such as stacking and grasping, on unseen complex geometries.
- Dexterous Manipulation: Similarly, Arthur Allshire et al. showed that reinforcement learning policies focusing on generalized contact points rather than shape-specific trajectories allow a multi-fingered robot to reorient diverse, unseen geometries in mid-air (Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger).
- Tactile Localization: In Learning Bayes Filter Models for Tactile Localization, Colin Keil et al. trained localization models on basic geometric primitives. The models successfully transferred to complex household objects because they learned the fundamental mathematical relationships between contact dynamics and local surface shape.
Modeling Space via Continuous Implicit Fields
A major bottleneck in geometric generalization is the choice of shape representation. Explicit representations, like 3D meshes or voxel grids, often force a network to conform to fixed resolutions or topologies, making it difficult to adapt to novel structures.
To achieve zero-shot generalization, recent architectures leverage continuous implicit representations, such as Signed Distance Functions (SDFs) or Unsigned Distance Fields (UDFs). Instead of asking "what category does this surface belong to?", the network evaluates the continuous space surrounding the object, mapping any given 3D coordinate to its proximity to the nearest interaction boundary.
In Grasping Field: Learning Implicit Representations for Human Grasps, Korrawe Karunratanakul et al. introduced the "Grasping Field," which models human-object interaction by mapping 3D points to their distances from both the hand and the object. By shifting the focus from explicit surface tracking to the continuous space between interacting entities, the model naturally generalizes to novel object geometries without requiring retraining.
Explicit Mesh Representation: [Vertices/Faces] ──► Rigidly tied to trained topologies
Implicit Field Representation: [3D Coordinate] ──► Distance to Boundary ──► Category-Agnostic
Grounding distance fields in explicit geometric physics further hardens them against distribution shifts. For example, Junhui Hou et al. developed GeoUDF, a framework that incorporates explicit geometric primitives into neural distance field estimations. By anchoring the network's predictions in mathematical principles rather than data memorization, the system achieves superior zero-shot surface reconstruction for open boundaries and complex interior structures across entirely unseen categories.
Similarly, meta-learning approaches can be used to build robust geometric priors. In GenSDF, Gene Chou et al. utilized a two-stage training framework that meta-learns on a mixture of labeled and unlabeled point clouds. This allows the model to capture a universal geometric prior, enabling high-fidelity 3D reconstruction across more than 100 unseen object classes in a zero-shot setting.
Decoupling Semantics from Spatial Reasoning
Traditional category-level estimation models suffer from "appearance bias"—they confuse what an object is with how it occupies space. If a model relies on category templates, it cannot process an object for which no template exists. Zero-shot geometric generalization circumvents this by strictly decoupling semantic recognition from geometric reasoning.
A clear execution of this strategy is found in Semantic Abstraction, where Huy Ha et al. separated semantic understanding from 3D spatial reasoning. They utilized a 2D Vision-Language Model (VLM) to generate semantic relevancy maps, which were then projected into a 3D module that predicted spatial occupancy. Because the 3D module only evaluates class-agnostic occupancy maps rather than explicit object identities, it learns universal geometric principles—such as shape completion and spatial relationships—that generalize seamlessly to rare or novel objects.
[Input Image] ──► 2D VLM (Identifies "What") ──► [Class-Agnostic Relevancy Map]
│
▼
3D Module (Reasons "Where") ──► Zero-Shot Occupancy
Conditioning spatial tasks directly on 3D geometry rather than category labels allows models to treat shapes as visual queries. In Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects, Xuchong Qiu et al. fed both an RGB image and an arbitrary 3D model into a neural network to estimate orientation relative to the shape itself. By avoiding learned category templates entirely, the system achieves high-precision zero-shot pose estimation for novel objects.
When explicit CAD models are unavailable at test time, models can instead leverage abstract shape priors or learn fundamental geometric invariants:
- Abstract Shape Priors: In CATRE, Xingyu Liu demonstrated that category-level pose refinement can be successfully executed by aligning partially observed point clouds with an abstract "mean shape" prior, proving that generalized category context is sufficient for neural refinement without exact CAD models.
- Intrinsic Geometric Properties: Rather than relying on semantic features, Jihyun Lee et al. trained a network to predict the Shape Laplacian—an intrinsic geometric property—directly from point clouds (Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian). This focus on volumetric geometry allowed the model to deform images of entirely unseen object categories without artifacting.
- Particle Hierarchies: For dynamic physical predictions, Elias Wang introduced the Hierarchical Relation Network in Flexible neural representation for physics prediction. By representing objects as a hierarchy of particles and learning local interaction rules, the model bypasses global shape templates to predict the complex physical dynamics of novel, non-convex rigid bodies and soft materials in a zero-shot manner.
By shifting the paradigm away from category-specific memorization toward local contact physics, continuous implicit fields, and semantic decoupling, these methods allow intelligent systems to navigate, manipulate, and reconstruct the open world with the same geometric flexibility as humans.