Factorized Representations for Compositional Generalization

42 concepts37 papers1 proposal

Human intelligence possesses a remarkable ability: we can effortlessly understand or imagine entirely novel combinations of familiar concepts. If you know what a "beaver" looks like and you know what "chocolate" is, you can instantly conceptualize a "chocolate beaver" despite never having encountered one. We achieve this through compositional generalization—the capacity to decompose the world into discrete, reusable primitives and recombine them flexibly.

Standard deep learning models, however, struggle with this type of out-of-distribution generalization. When a traditional neural network learns to recognize a "red ball," it often entangles the features of "redness" and "roundness" into a single joint representation. If it is later presented with a "red cube" or a "blue ball" during testing, it frequently fails because it has memorized specific co-occurrence patterns rather than isolating the underlying factors of variation.

As Mohammad Hossein Rohban et al. demonstrate in Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Models, explicit representation disentanglement is the primary driver behind a model's capacity for compositional generalization. To build models that scale efficiently without requiring exhaustive training on every possible combination, we must move away from monolithic architectures. As Yilun Du et al. argue in Position: Compositional Generative Modeling: A Single Model is Not All You Need, constructing complex systems by combining smaller, independent generative factors allows for combinatorial extrapolation that monolithic joint distributions simply cannot achieve.

The Geometry of Independence: Subspaces and Slots

To understand why factorized representations work, it helps to look at the underlying geometry of the latent space. If a model maps concepts to a single, high-dimensional vector space without constraints, different attributes will interfere with one another. Factorization solves this by creating independent pathways or orthogonal subspaces for different properties.

Entangled Representation (Monolithic)      Factorized Representation (Subspaces)
       [ Red Ball ]                             Color Subspace      Shape Subspace
            │                                         │                   │
  (Features are mixed;                        [Red] ──┴───┐       [Ball] ─┴───┐
   cannot cleanly swap                          │         │         │         │
   "Red" for "Blue")                          [Blue] ─────┼─────> [Cube] ─────┴─> Zero-Shot
                                                          │                       Combination
                                                          └─────────────────────> [Blue Cube]

One elegant way to enforce this geometric separation is through structural orthogonality. In Attention-based Iterative Decomposition for Tensor Product Representation, Taewon Park et al. show that maintaining high orthogonality between structural "roles" and "fillers" allows a network to decompose and recombine symbolic sequences without mutual interference. This architectural choice mirrors biological systems; Michael G. Müller et al. propose in A Model for Structured Information Representation in Neural Networks of the Brain that the human brain relies on factorizing abstract roles and sensory content to perform dynamic variable binding.

In computer vision, this geometric separation is often implemented via multi-path or slot-based architectures:

Decoupling Objects from Attributes and Relations

By isolating primitives into independent representations, models can perform Compositional Zero-Shot Learning (CZSL)—recognizing unseen pairs of known attributes and objects.

When a model is forced to predict attributes as independent primitives rather than joint embeddings, it acts as a powerful regularizer. As Massimiliano Mancini et al. demonstrate in KG-SP: Knowledge Guided Simple Primitives for Open World Compositional Zero-Shot Learning, using separate feature extractors for distinct properties like shape and texture prevents the model from overfitting to specific seen combinations. This decoupling allows for substantial increases in accuracy on unseen compositions without requiring exhaustive training data, a property also leveraged by Youssef Zahran et al. in Anticipating Future Object Compositions without Forgetting to recognize novel attribute-object pairs while preventing catastrophic forgetting.

This factorization principle extends beyond static visual attributes to relational and action-based contexts:

Decoupling Dynamics and Interaction Rules

Compositional generalization is equally vital for understanding physical laws and dynamics. If a model entangles an object's visual appearance with the rules governing its behavior, it cannot predict how a newly encountered object will interact with its environment.

To decouple "what" an object is from "how" it behaves, Aniket Didolkar et al. introduce Neural Production Systems. By separating the definition of an interaction rule from the specific properties of the entities involved, their architecture can apply a learned behavior—such as a collision rule—to novel object types without additional training.

Holistic Physics Model                     Compositional Law Parsing
┌──────────────────────────────┐           ┌──────────────────┐     ┌──────────────────┐
│ Object Appearance + Dynamics │           │  Object Latents  │     │  Latent Physics  │
│ (Overfits to training shapes)│           │ (Shape, Texture) │     │ (Collision, Gravity)│
└──────────────────────────────┘           └────────┬─────────┘     └────────┬─────────┘
                                                    └───────────┬────────────┘
                                                                ▼
                                                    Zero-Shot Dynamics Prediction
                                                    on Unseen Object Appearances

Similarly, the authors of Compositional Law Parsing with Latent Random Functions separate static object attributes from the latent functions governing their motion. Because the learned rules are independent of appearance features, the model can accurately predict the dynamics of objects with entirely unseen visual properties. This form of operational factorization is also highly effective for learning environmental transformations; Barthélémy Dang-Nhu et al. demonstrate in Disentangled representation learning through unsupervised symmetry group discovery that discovering independent symmetry groups for actions allows an agent to correctly predict the effects of transformations in completely novel configurations.

Training Strategies for Inducing Disentanglement

How do we force a neural network to learn these factorized representations without relying on exhaustive manual annotations? Researchers have developed several distinct training paradigms to incentivize independence:

Unsupervised and Structured Generative Modeling

One foundational approach is to embed statistical constraints directly into generative architectures. Xander Steenbrugge et al. demonstrate in Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations that using an unsupervised Variational Autoencoder (VAE) to pre-learn independent generative factors (like shape and position) provides a robust foundation for solving abstract reasoning tasks. To isolate interactions cleanly, Kaspar Märtens et al. introduce Neural Decomposition: Functional ANOVA with Variational Autoencoders, which embeds a functional ANOVA structure into a VAE. By isolating specific marginal effects from interaction terms, the model can accurately synthesize data points representing attribute pairings that were entirely absent during training.

Weak Supervision and Environmental Robustness

When full unsupervised separation is difficult, weak supervision can guide latent alignment. Yordan Hristov et al. show in Interpretable Latent Spaces for Learning from Demonstration that using weak supervision to untangle latent dimensions along human concepts allows robots to ground abstract instructions into real-world objects and recognize novel combinations. Furthermore, this factorization provides environmental resilience; Nick Pawlowski et al. demonstrate in Representation Disentanglement for Multi-task Learning with application to Fetal Ultrasound that explicitly separating core anatomical features from environmental artifacts allows models to maintain high accuracy when familiar features appear in novel, out-of-distribution contexts.

Predictive Objectives and Compositional Sampling

Disentanglement can also emerge naturally from predictive tasks or structured data sampling:

  • Predictive World Models: In Probing Emergent Semantics in Predictive Agents via Question Answering, Abhishek Das et al. discover that agents trained purely to predict future sensory observations spontaneously develop factorized internal representations of color, shape, and position, unlocking emergent compositional reasoning without explicit labels.
  • Semi-Positive Compositional Sampling: To explicitly prevent a model from memorizing fixed triplets during training, Fang Nan et al. propose a training strategy in Visual-guided human-object interaction detection using semi-positive examples. By sourcing different components of a target relationship (such as the action and the object) from separate images, the model is forced to learn independent representations for each primitive, enabling it to recognize novel combinations it has never seen as a single unit.

Generalization Beyond Vision

While visual attribute composition is an intuitive sandbox, factorized representations are equally powerful in non-visual domains.

In natural language processing, traditional sentiment models often struggle when multiple entities and aspects are discussed simultaneously. To resolve this, Jun Yang et al. develop the CEA method in Multi-Entity Aspect-Based Sentiment Analysis With Context, Entity and Aspect Memory. By learning independent semantic representations for entities and aspects before combining them, the model successfully predicts sentiment for specific pairs it never encountered during training, bypassing memorized co-occurrence patterns.

In spatiotemporal forecasting, models often fail when data is fragmented or when environmental conditions shift unexpectedly. Zhengyang Zhou et al. address this in Towards Learning in Grey Spatiotemporal Systems: A Prophet to Non-consecutive Spatiotemporal Dynamics by decoupling environmental factors from historical spatial observations. Because these predictive influences are separated into distinct components, the model generalizes seamlessly to novel combinations of conditions (such as an unobserved weather pattern occurring on a specific day of the week), ensuring robust predictions even when recent data is missing.

Why It Works: Beating the Combinatorial Explosion

Ultimately, factorized representation learning works because it fundamentally alters how a model scales with data complexity.

Consider a domain with $N$ attributes and $M$ objects. A monolithic model that attempts to learn the joint distribution $P(\text{attribute}, \text{object})$ must see a significant fraction of the $N \times M$ combinations to generalize reliably. As the number of primitives grows, the data requirement suffers from an exponential combinatorial explosion.

Factorization breaks this curse of dimensionality. By constraining the architecture to learn independent distributions—$P(\text{attribute})$ and $P(\text{object})$—and combining them via a deterministic, structural operator, the model only needs to see each primitive a linear number of times. Generalization to an unseen combination ceases to be an extrapolation problem; instead, it becomes a simple, zero-shot operation of re-composing independent coordinates within a structured latent space.

Key Foundational Work

Multi-Entity Aspect-Based Sentiment Analysis With Context, Entity and Aspect Memory (2018) — contributed 2 concept elements to this direction.
Interpretable Latent Spaces for Learning from Demonstration (2018) — contributed 1 concept elements to this direction.
Learning Compositional Sparse Models of Bimodal Percepts (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_generalization_combinatori_09d2b8a6,
  title = {compositional generalization (combinatorial generalization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-combinatorial-generalization/rdr_09d2b8a68a8a4363ac6a48e3069fc2e6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_compositional_generalization_combinatori_09d2b8a6_v1,
  title = {compositional generalization (combinatorial generalization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-combinatorial-generalization/rdr_09d2b8a68a8a4363ac6a48e3069fc2e6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake