Generalizing Robotic Interaction to Novel Object Categories

40 concepts37 papers1 proposal

Traditional robotic systems operate under a "closed-world" assumption. If a robot is trained to detect, track, or grasp a specific set of items, it performs exceptionally well within that narrow distribution. However, when deployed in dynamic, human-centric environments, this rigid paradigm breaks down. A robot tasked with navigating a home or clearing a table will inevitably encounter objects outside its training set—such as novel tools, unmodeled obstacles, or unique household items.

As noted by Leibe et al., conventional tracking-by-detection systems are fundamentally constrained by the predefined categories of their underlying classifiers. To operate effectively in the real world, robots must shift from recognizing specific learned instances to understanding broad, open-set categories—a challenge highlighted early on by Zender et al..

To achieve zero-shot generalization to novel object categories, researchers are moving away from rigid taxonomic labels. Instead, they are decomposing objects into three core representations: semantic attributes, functional affordances, and geometric correspondences.


Deconstructing Categories into Semantic Attributes

Why does a human know how to interact with an unfamiliar object? Often, it is because we can describe its properties using language. If a robot can break down a monolithic, unseen category into a composition of known visual and textual properties, it can interact with entirely new items.

Yang et al. (2021) and later Yang et al. (2025) demonstrated this intuition by building robotic grasping systems that identify targets using natural language attributes (e.g., "red sphere") rather than fixed category labels. By combining multimodal learning with data-efficient adaptation techniques like one-grasp fine-tuning, the authors showed that a robot can successfully isolate and grasp novel items in cluttered environments because the underlying attributes remain consistent across categories.

[Rigid Classification]  Unseen Object ──> "Unknown Category" ──> Failure
[Attribute-Based]       Unseen Object ──> [Red] + [Sphere]   ──> Successful Grasp

The explosion of vision-language foundation models has scaled this capability significantly. Rather than training custom attribute classifiers, researchers can leverage the embedded semantic knowledge of large models to bridge the gap between an object's physical shape and an intended action.

  • GraspGPT: Tang et al. (2023) introduced a framework that uses Large Language Models (LLMs) as a zero-shot semantic bridge, allowing a robot to deduce how to grasp an unfamiliar object based on its textual description and contextual task requirements.
  • GraspCLIP: Similarly, Tang et al. (2023) utilized CLIP's joint vision-language embeddings to generalize task-specific grasping logic (such as holding a knife by the handle to cut, versus holding it by the blade to hand it over) across entirely unseen object categories.
  • Instruction-Driven Routing: To handle highly ambiguous commands and out-of-distribution scenarios, Li et al. developed CogVLA, a vision-language-action model that mimics human cognitive pathways to map vague commands to precise physical actions.

This language-driven generalization also applies to spatial reasoning. For instance, Blukis et al. showed that a robot can follow navigation instructions involving novel items at test time simply by grounding the commands using a few reference photos and descriptions, bypassing the need for explicit category pre-training.


Grounding Actions in Functional Affordances

An object's semantic name matters less than what it does. A knife, a shard of glass, and a pizza cutter look entirely different, yet they all share the functional affordance of "cutting." By training robots to perceive the world through the lens of potential actions rather than object classes, zero-shot generalization happens naturally.

Nguyen et al. built on this concept by learning a joint embedding space for natural language verbs and RGB images. This allows a robot to retrieve tools it has never seen before based purely on functional usage queries (e.g., "something to cut with"). The system bypasses the category bottleneck by focusing on the visual features that correlate with the physical action.

For articulated objects like cabinets, drawers, or appliances, the challenge is identifying which parts move. Zhang et al. developed IAAO, a framework that leverages 3D Gaussian Splatting and foundation models to predict action affordances—such as identifying a handle for pulling or a hinge for swinging—without requiring category-specific training.

Unseen Articulated Object ──> 3D Gaussian Splatting ──> Identify Handle Geometry ──> Pull Action

When the task shifts from simple grasping to complex arrangement, generative models can infer valid placement affordances. Chang et al. utilized a diffusion model to identify valid "placeable regions" for novel object storage tasks. Taking this a step further, Weiyu Liu introduced StructDiffusion, which combines diffusion models with object-centric transformers. This allows a robot to manipulate and arrange entirely unseen geometries into physically valid structures (like a table setting or a tower) based solely on high-level language instructions, because the model learns the underlying physics of structural stability rather than rigid object classes.


Geometric Correspondence and Few-Shot Alignment

If a robot cannot rely on semantic labels or functional descriptions, it can fall back on pure geometry. If the system can map the structural topology of a novel object to a known reference shape, it can transfer manipulation skills without needing to know what the object is.

Qiu et al. demonstrated that by feeding both an RGB image and a 3D shape model into a deep network, a system can estimate the 3D orientation of completely novel categories. The network learns to relate orientation to geometric shape rather than semantic features, enabling zero-shot pose estimation.

This geometric alignment is highly effective for imitation learning. To transfer a manipulation skill (like pouring or sweeping) from a single human demonstration to an entirely new object class, Johns et al. proposed a Retrieval-Alignment-Replay pipeline. The system finds a visually similar known object, calculates the spatial alignment between the new and old shapes, and replays the trajectory. Vosylius et al. generalized this concept by framing imitation learning as an energy-based graph alignment problem, allowing a robot to map task-specific trajectories across diverse geometries without task-specific fine-tuning.

Demonstration (Known Object) ──┐
                                ├─> Graph/Point Cloud Alignment ─> Executed Trajectory
Novel Object Geometry ──────────┘

Geometric abstractions also allow robots to extrapolate tasks to more complex scenes. Huang et al. introduced VIRL, a self-supervised framework that uses visual graph abstractions to help robots manipulate a larger number of objects than they saw during training. In cluttered environments, Yu et al. combined a one-shot similarity network with reinforcement learning to search for and grasp novel items hidden completely under piles, relying on visual similarity rather than explicit classification.

Crucially, context can guide these geometric interactions. Ramachandruni et al. developed ConSOR to infer how to rearrange novel objects by reading contextual cues from partially organized scenes, determining where an unseen item belongs based on its spatial relationship to surrounding objects.

While deep learning dominates open-world perception, pure spatial reasoning offers a powerful, data-efficient alternative. Hu et al. demonstrated that geometric point cloud modeling can outperform standard deep learning models when adapting to unseen everyday objects for soft hand exoskeletons, achieving high success rates without massive datasets.

This shift away from narrow, instance-level perception is echoed across the field: from the real-time semantic SLAM pipelines developed by De Gregorio et al. that prioritize category-level understanding over specific instances, to the agile locomotion strategies proposed by Zhao et al. that enable quadrupedal robots to track and interact with arbitrary targets against complex backgrounds. By anchoring perception in geometry, attributes, and affordances, modern robotics is successfully moving past the closed-world bottleneck.

Key Foundational Work

Taking mobile multi-object tracking to the next level (2012) — contributed 1 concept elements to this direction.
An Integrated Robotic System for Spatial Understanding and Situated Interaction in Indoor Environments (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_zero_shot_object_generalization_8545a833,
  title = {zero-shot object generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-object-generalization/rdr_8545a83373e44cf4b6419ee28868158f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_zero_shot_object_generalization_8545a833_v1,
  title = {zero-shot object generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-object-generalization/rdr_8545a83373e44cf4b6419ee28868158f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake