Compositional Zero-Shot Learning for Visual Concepts

31 concepts29 papers4 proposals

Humans possess a remarkable cognitive ability: if we know what a "cat" looks like and we understand the adjective "young," we can instantly recognize a "young tiger" the first time we encounter one, even if we have only ever seen adult tigers and young cats. This ability to systematically combine known primitives to understand entirely novel scenarios is called compositional generalization.

In contrast, standard deep learning models are notorious for relying on superficial shortcuts. If a model is trained exclusively on "red apples" and "yellow bananas," it frequently fails to identify a "red banana." Instead of treating visual concepts as combinations of independent primitives (such as states, attributes, objects, or parts), traditional neural networks tend to memorize training combinations as monolithic entities.

To expose and measure this limitation, researchers have designed rigorous evaluation benchmarks. For instance, Aishwarya Agrawal et al. introduced C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset to test whether models can answer questions about novel combinations of concepts seen during training. Similarly, Vignesh Ramanathan developed PACO: Parts and Attributes of Common Objects, a fine-grained dataset highlighting the massive performance gap that occurs when models must compose independent object parts and attributes without explicit joint training examples.

The goal of Compositional Zero-Shot Learning (CZSL) is to bridge this gap, enabling vision systems to recognize unseen combinations of known primitives.

The Fundamental Tension: Disentanglement vs. Context

The most intuitive way to solve CZSL is through complete disentanglement. If a network can extract an attribute feature (like "purple") completely independently of the object feature (like "cube"), it can simply recombine these features at inference time to recognize a novel pair.

Early approaches leaned heavily into this modular philosophy. For example, Justin Johnson et al. demonstrated in Inferring and Executing Programs for Visual Reasoning that translating visual reasoning tasks into explicit logical programs executed by a dynamic assembly of neural modules significantly improves generalization to novel concepts. In a similar vein of modularity, Frank Ruis et al. developed ProtoProp in Independent Prototype Propagation for Zero-Shot Compositionality, which treats visual features as independent building blocks and propagates them through a graph to decouple attributes from objects entirely.

However, pure disentanglement runs into a major physical reality: attributes are context-dependent. A "sliced tomato" looks fundamentally different from a "sliced loaf of bread." The visual manifestation of an attribute changes depending on the object it modifies. If a model learns a single, rigid prototype for "sliced," it will fail to generalize across highly diverse object categories.

Attributes as Operators and Transformations

To resolve this tension, researchers shifted from viewing attributes as static labels to viewing them as operators or transformations.

In a foundational shift, Tushar Nagarajan et al. proposed in Attributes as Operators: Factorizing Unseen Attribute-Object Compositions that attributes should be modeled as mathematical matrices that act upon object vectors. Instead of trying to find a static visual representation for "rusty," the model learns how the "rusty" operator transforms the base representation of an object like a "car" or an "anchor."

This functional perspective has been formalized using advanced mathematical frameworks. For instance, Sotirios Panagiotis Chytas et al. introduced CatCom in Understanding Multi-compositional Learning in Vision and Language Models via Category Theory, applying Category Theory to treat attributes as "morphisms"—structural transformations rather than static objects. This allows the model to map how properties dynamically alter the underlying geometry of the object space, preserving context while maintaining the ability to generalize.

Dynamic and Conditional Representations

Another strategy to handle context-dependency is to allow the model to learn multiple or conditional representations of a single primitive, explicitly accounting for the object's context on the fly.

Instead of a one-size-fits-all embedding, Lingqiao Liu et al. developed CANet in Learning Conditional Attributes for Compositional Zero-Shot Learning. CANet uses a hyper-learner to generate custom attribute embeddings dynamically, directly conditioned on the specific object and image context. This ensures that the representation of "wet" adapts appropriately whether it modifies an apple or a cat.

Similarly, Muli Yang et al. proposed the HiDC framework in Learning Unseen Concepts via Hierarchical Decomposition and Composition, which breaks images down into attribute and object subspaces before adaptively recombining them using "semi-positive" samples to capture subtle appearance shifts. To handle the sheer diversity of meanings, Dongyao Jiang et al. introduced MRSP in MRSP: Learn Multi-representations of Single Primitive for Compositional Zero-Shot Learning, using cross-attention and Graph Convolutional Networks to learn a union of multiple representations for each primitive, successfully distinguishing between contexts like a "historical town" and a "senior tiger."

Language-Guided Compositionality

With the rise of large vision-language models like CLIP, language has emerged as a powerful regularizer for compositionality. Because text naturally factorizes concepts into discrete words, it provides a semantic blueprint that can guide visual models.

For instance, Nihal V. Nayak et al. introduced Compositional Soft Prompting (CSP) in Learning to Compose Soft Prompts for Compositional Zero-Shot Learning. Instead of tuning an entire vision backbone, CSP treats attributes and objects as learnable "soft" tokens in the text encoder that can be dynamically recombined. Following this, Henan Wang et al. proposed Hierarchical Prompt Learning (HPL) in Hierarchical Prompt Learning for Compositional Zero-Shot Recognition, deploying specialized prompt experts to explicitly disentangle states from objects within CLIP's latent space.

To inject even richer semantic knowledge, Wentao Bao et al. developed PLID in Prompting Language-Informed Distribution for Compositional Zero-Shot Learning. PLID leverages Large Language Models to generate diverse, descriptive text distributions for visual concepts, helping the model bridge the gap between individual primitive recognition and holistic composition by grounding visual features in rich linguistic descriptions.

Why It Works: The Intuition

Compositional Zero-Shot Learning works because it forces neural networks to move away from memorizing holistic image templates and toward learning generative rules of composition. Whether by treating attributes as mathematical operators, dynamically conditioning them on object context, or using language prompts to enforce structural boundaries, these methods prevent the model from overfitting to training pairs. By mastering how components interact rather than just what they look like in isolation, vision systems achieve the flexibility required to navigate an open, unpredictable world.

Key Foundational Work

Learning Compositional Sparse Models of Bimodal Percepts (2014) — contributed 1 concept elements to this direction.
C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset (2017) — contributed 1 concept elements to this direction.
Inferring and Executing Programs for Visual Reasoning (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_generalization_composition_2afa31ed,
  title = {compositional generalization (compositional zero-shot learning)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-compositional-zero-shot-learning/rdr_2afa31edefd240f9b64b5196bdc82045},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_compositional_generalization_composition_2afa31ed_v1,
  title = {compositional generalization (compositional zero-shot learning)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-compositional-zero-shot-learning/rdr_2afa31edefd240f9b64b5196bdc82045/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake