Compositional Generalization in Generative Models

19 concepts19 papers1 proposal

Humans possess a remarkable capacity for "visually grounded imagination": if you know what a "smiling man" and a "neutral woman" look like, you can effortlessly visualize a "smiling woman" despite never having seen one. However, standard generative models like Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) frequently fail this test. As noted by Chrysos et al., these models struggle with compositional generalization—the ability to synthesize novel combinations of attributes that were never observed together during training.

This "combinatorial gap" arises because models often entangle independent features in their latent space. When a model learns to generate a specific object, it often binds the object's identity to the specific colors, shapes, or textures present in the training data. To overcome this, researchers are moving away from monolithic end-to-end learning toward architectures that treat concepts as modular, recombinable units.

Modular and Object-Centric Representations

One intuitive solution is to force the model to see the world as a collection of independent parts. Wiedemer et al. provide a theoretical foundation for this, arguing that representing scenes as a collection of independent "slots" or objects is the key to generalizing to new arrangements. By isolating objects into separate latent variables, the model can rearrange them like building blocks.

This modularity can also be achieved through "neural programs." Deng et al. proposed PNP-Net, which uses specialized probabilistic operators to compose complex scenes from text descriptions. Similarly, Carmeli et al. introduced a "Composition through Decomposition" (CtD) regime. Their insight is that to build complex ideas, a model must first learn to take them apart. By decomposing images into a discrete codebook of basic concepts, agents can later perform zero-shot composition of novel descriptions.

Logical Composition via Energy Functions

If concepts are treated as independent probability distributions, we can use the language of logic to combine them. Li et al. demonstrated that Energy-Based Models (EBMs) are particularly well-suited for this. In an EBM, a concept is represented by an energy function where low energy indicates high probability. By applying logical operators—AND, OR, and NOT—directly to these energy functions, the authors enable the zero-shot creation of scenes that satisfy multiple constraints simultaneously.

Building on this, Nie et al. developed LACE, which performs this energy-based composition within the latent space of pre-trained generators like StyleGAN. This allows for high-resolution, multi-attribute editing without retraining the base model. Similarly, Vedantam et al. used a Product-of-Experts (PoE) approach, where different "expert" networks focus on specific attributes, and their combined inference allows the model to navigate a hierarchy of abstraction to compose unseen pairings.

The Geometry of Transformations

Rather than just learning what an attribute is, some researchers argue models should learn what an attribute does. Hwang et al. introduced MAGANet, which models attributes as mathematical "group actions" or transformations. Instead of representing a "red car" as a static point, the model learns the transformation that "moves" any object toward the property of being red. This geometric approach allows for superior combinatorial generalization because the transformation for "red" can be applied to any object, even those never seen in red during training.

Ge et al. achieved a similar effect through Group-Supervised Learning (GSL). By explicitly swapping latent features between groups of related images (e.g., taking the color from a boat and applying it to a car), their GZS-Net forces the model to disentangle attributes, enabling high-fidelity zero-shot synthesis.

Emergence and Data Distributions

Recent evidence suggests that compositional ability might be an emergent property of how models learn. The authors of recent work on diffusion models found that complex generalizations appear suddenly only after a model has mastered all constituent "atomic" skills. This suggests a multiplicative relationship: the ability to combine concepts is not learned linearly but emerges once the underlying concept graph is sufficiently populated.

Furthermore, the nature of the training data itself matters. Farid et al. identified that training on continuous distributions rather than discrete categories is a "secret sauce" for generalization. By adding a continuous, self-supervised auxiliary loss, models become significantly better at mixing and matching attributes like color, shape, and size in out-of-distribution scenarios.

Whether through explicit modularity, logical energy functions, or geometric transformations, the goal remains the same: moving generative AI from simple imitation to a flexible, human-like capacity for combinatorial synthesis.

Key Foundational Work

Probabilistic Neural Programmed Networks for Scene Generation (2018) — contributed 1 concept elements to this direction.
Multilinear Latent Conditioning for Generating Unseen Attribute Combinations (2020) — contributed 1 concept elements to this direction.
Generative Models of Visually Grounded Imagination (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_generalization_combinatori_f873078f,
  title = {compositional generalization (combinatorial out-of-distribution generalization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-combinatorial-out-of-distribution-generalization/rdr_f873078fc40645c4851c98c223cc1eb4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_compositional_generalization_combinatori_f873078f_v1,
  title = {compositional generalization (combinatorial out-of-distribution generalization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-combinatorial-out-of-distribution-generalization/rdr_f873078fc40645c4851c98c223cc1eb4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake