Relational Structure for Zero-Shot Generalization

67 concepts59 papers1 proposal

Imagine trying to recognize a quokka having never seen an image of one before. If you are simply given an isolated text embedding for the word "quokka," the task is incredibly difficult. Traditional Zero-Shot Learning (ZSL) attempts to bridge this gap by learning a direct mapping from visual features to independent semantic vectors. However, this approach frequently suffers from a "projection domain shift"—the mapping learned for familiar classes fails to generalize to unseen ones because it treats every category as an isolated island.

But humans do not encounter concepts in a vacuum. If you are told that a quokka is a marsupial, closely related to kangaroos and wallabies, and shares structural similarities with small macropods, you can immediately form a mental prototype.

The core insight of relational zero-shot generalization is that unseen categories are not completely alien; they are simply unobserved nodes in a deeply interconnected network of knowledge. By explicitly modeling the relational structure—such as taxonomies, knowledge graphs, or scene graphs—that connects known and unknown concepts, machine learning models can infer the properties, visual features, or classifier weights of entirely novel categories based on their topological neighborhood.


Anchoring Unseen Concepts in Taxonomic Manifolds

To recognize something new, a model must first understand where that concept sits relative to what it already knows. Instead of treating classification labels as flat, equidistant choices, relational generalization embeds entire hierarchical structures directly into the model's latent space.

Mettes et al. demonstrate the power of this approach by embedding pre-defined action taxonomies into a shared hyperbolic space. Because hyperbolic geometry naturally preserves tree-like hierarchical distances, positioning a novel class relative to known actions allows the model to perform zero-shot retrieval without requiring specific training examples.

 Flat Label Space (Traditional)           Taxonomic Manifold (Relational)
    [Dog]   [Cat]   [Quokka]                         [Mammal]
      |       |        |                            /        \
   (Equidistant Vectors)                    [Carnivora]    [Marsupial]
                                             /      \        /      \
                                          [Dog]   [Cat] [Kangaroo] [Quokka]
                                                                      *Unseen*

When the internal feature space preserves these hierarchical distances, the model reserves specific regions for broader categories. As Zeng et al. show, enforcing this structural alignment allows a model to accurately categorize previously unseen specific instances by placing them within their correct taxonomic context.

This principle extends far beyond standard computer vision. In transcriptomics, Han et al. demonstrate that learning the underlying manifold of a biological cell ontology enables the zero-shot classification of completely unseen cell types based on their relational distance to known relatives. By leveraging the logical connectivity of an ontology, as Yu et al. observe, hierarchical models can assign labels to nodes that have zero associated instances in the training data, effectively discovering "orphan" terms purely through their structural relationships.


Knowledge Propagation via Graph Neural Networks

If a taxonomy provides the map, how does a model actually synthesize a classifier for a node it has never seen? The mechanism relies on graph-based knowledge propagation.

Instead of mapping classes to features independently, Peike Li et al. demonstrate that processing category relationships collectively via graphs allows a model to utilize the boundaries and characteristics of known classes to better define the feature space for novel categories.

To operationalize this, researchers use Graph Convolutional Networks (GCNs) to map semantic embeddings from a knowledge graph directly into the weight space of a visual classifier.

  1. Graph Construction: A knowledge graph (like WordNet) connects seen and unseen classes via semantic edges (e.g., is-a, has-part).
  2. Feature Aggregation: The GCN passes messages along these edges, allowing unseen nodes to aggregate semantic and visual priors from their seen neighbors.
  3. Weight Synthesis: The network outputs a predicted visual classifier weight vector for the unseen class.

Zechao Li et al. use this exact pipeline to infer the parameters of a visual classifier for a novel class without extensive visual training data. Similarly, Bo Jin et al. leverage GNNs to model topological relationships between classes, aggregating features from semantically related neighbors to predict the location of novel categories in the visual space.

To make this scaling practical for complex actions, Luo et al. apply GCNs to separate semantic hierarchies for verbs and nouns. By propagating knowledge through distinct relational streams, the system can recognize novel human-object interactions linearly rather than quadratically as vocabularies grow.


Shifting from Entity Memorization to Structural Invariants

A deeper facet of this research direction is moving entirely away from specific entity identifiers toward learning universal relational patterns. If a model merely memorizes what a specific entity means, it fails when encountering a completely new entity type.

To solve this "cold-start" problem, Zhou et al. propose an entity-agnostic representation learning framework. By prioritizing the relational structure of a knowledge graph over specific entity IDs, the system learns how categories of information interact. This allows it to apply learned interaction patterns to novel subjects seamlessly.

  Entity-Specific (Fails on Unseen)         Entity-Agnostic (Generalizes)
    [Entity: 402] --(Likes)--> [Entity: 11]     (Node_Type_A) --(Rel_X)--> (Node_Type_B)
          |                          |                |                          |
   (Must be seen in training)                  (Applies to any novel A and B sharing this structure)

This concept reaches its logical conclusion in fully inductive link prediction on hypergraphs. Huang et al. demonstrate that models can generalize to entirely novel relation types by learning the structural invariants of how entities are shared across different positions. This enables zero-shot performance even when none of the relations encountered at inference time were present during training. As Bauer et al. show, focusing on the structural relationships between pairs of entities rather than individual properties allows graph motifs learned in one domain to generalize to novel stimuli.


Collective and Contextual Inference

Why resolve zero-shot classification one isolated instance at a time? Real-world data provides context, and that context can itself be modeled as a local graph to resolve ambiguity.

Instead of performing independent dot-product comparisons between an image and a label, Liangliang Shi et al. frame zero-shot classification as a collective graph matching task over an entire batch. By considering the global context of similarities between all samples and labels simultaneously, the model utilizes batch-level context to refine individual predictions, moving beyond isolated similarity scores.

On an instance level, objects rarely appear in isolation. Hengcan Shi et al. leverage scene graphs to discover novel categories through linguistic cues in known relational structures. For example, if a model encounters an unfamiliar object in the context of a "man riding [X]" relationship, the familiar subject and predicate allow it to infer the presence and location of the unfamiliar object.

Ultimately, as argued by Hui Nie et al., prioritizing the relationships between visual regions and semantic labels over individual feature refinement enhances a model's ability to generalize. The connections between concepts are inherently more stable and informative for zero-shot tasks than isolated feature vectors, providing the structural scaffolding necessary for true open-vocabulary understanding.

Key Foundational Work

Predicting protein function via downward random walks on a gene ontology (2015) — contributed 2 concept elements to this direction.
Metric Learning for Large Scale Image Classification: Generalizing to New Classes at Near-Zero Cost (2012) — contributed 1 concept elements to this direction.
Learning semantic relationships for better action retrieval in images (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_relational_zero_shot_generalization_d9f39123,
  title = {relational zero-shot generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relational-zero-shot-generalization/rdr_d9f39123a5f94cccad207210ade1adfc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_relational_zero_shot_generalization_d9f39123_v1,
  title = {relational zero-shot generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relational-zero-shot-generalization/rdr_d9f39123a5f94cccad207210ade1adfc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake