Human intelligence possesses a remarkable ability: we can effortlessly understand or imagine entirely novel combinations of familiar concepts. If you know what a "beaver" looks like and you know what "chocolate" is, you can instantly conceptualize a "chocolate beaver" despite never having encountered one. We achieve this through compositional generalization—the capacity to decompose the world into discrete, reusable primitives and recombine them flexibly.
Standard deep learning models, however, struggle with this type of out-of-distribution generalization. When a traditional neural network learns to recognize a "red ball," it often entangles the features of "redness" and "roundness" into a single joint representation. If it is later presented with a "red cube" or a "blue ball" during testing, it frequently fails because it has memorized specific co-occurrence patterns rather than isolating the underlying factors of variation.
As Mohammad Hossein Rohban et al. demonstrate in Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Models, explicit representation disentanglement is the primary driver behind a model's capacity for compositional generalization. To build models that scale efficiently without requiring exhaustive training on every possible combination, we must move away from monolithic architectures. As Yilun Du et al. argue in Position: Compositional Generative Modeling: A Single Model is Not All You Need, constructing complex systems by combining smaller, independent generative factors allows for combinatorial extrapolation that monolithic joint distributions simply cannot achieve.
The Geometry of Independence: Subspaces and Slots
To understand why factorized representations work, it helps to look at the underlying geometry of the latent space. If a model maps concepts to a single, high-dimensional vector space without constraints, different attributes will interfere with one another. Factorization solves this by creating independent pathways or orthogonal subspaces for different properties.
Entangled Representation (Monolithic) Factorized Representation (Subspaces)
[ Red Ball ] Color Subspace Shape Subspace
│ │ │
(Features are mixed; [Red] ──┴───┐ [Ball] ─┴───┐
cannot cleanly swap │ │ │ │
"Red" for "Blue") [Blue] ─────┼─────> [Cube] ─────┴─> Zero-Shot
│ Combination
└─────────────────────> [Blue Cube]
One elegant way to enforce this geometric separation is through structural orthogonality. In Attention-based Iterative Decomposition for Tensor Product Representation, Taewon Park et al. show that maintaining high orthogonality between structural "roles" and "fillers" allows a network to decompose and recombine symbolic sequences without mutual interference. This architectural choice mirrors biological systems; Michael G. Müller et al. propose in A Model for Structured Information Representation in Neural Networks of the Brain that the human brain relies on factorizing abstract roles and sensory content to perform dynamic variable binding.
In computer vision, this geometric separation is often implemented via multi-path or slot-based architectures:
- Multi-Path Routing: Instead of processing an input holistically, models can route different attributes through isolated branches. For instance, Siteng Huang introduces the Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning framework, which processes states, objects, and their compositions along three distinct paths, ensuring that primitive features are learned independently before being combined.
- Slot-Based Mixtures: Rather than compressing an entire scene into a single vector, the authors of Compositional Mixture Representations for Vision and Text represent scenes as a collection of individual constituent parts using Gaussian Mixture Models. This allows the model to extrapolate to novel object combinations by leveraging the learned properties of discrete components.
- 3D Attribute Factorization: In spatial environments, Sungbin Mun et al. develop a 3D-aware Disentangled Representation for Compositional Reinforcement Learning that factorizes visual scenes into distinct sub-vectors for color, shape, and 3D position. Similarly, Mihir Prabhudesai et al. show in Disentangling 3D Prototypical Networks for Few-Shot Concept Learning that decoupling 3D shape codes from 1D style codes enables a model to recognize or generate novel object configurations from minimal examples.
Decoupling Objects from Attributes and Relations
By isolating primitives into independent representations, models can perform Compositional Zero-Shot Learning (CZSL)—recognizing unseen pairs of known attributes and objects.
When a model is forced to predict attributes as independent primitives rather than joint embeddings, it acts as a powerful regularizer. As Massimiliano Mancini et al. demonstrate in KG-SP: Knowledge Guided Simple Primitives for Open World Compositional Zero-Shot Learning, using separate feature extractors for distinct properties like shape and texture prevents the model from overfitting to specific seen combinations. This decoupling allows for substantial increases in accuracy on unseen compositions without requiring exhaustive training data, a property also leveraged by Youssef Zahran et al. in Anticipating Future Object Compositions without Forgetting to recognize novel attribute-object pairs while preventing catastrophic forgetting.
This factorization principle extends beyond static visual attributes to relational and action-based contexts:
- Visual Relationships: In Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detection, Tim Salzmann et al. disentangle object and predicate embeddings, allowing the model to successfully compose and identify rare or unseen visual triplets (e.g., a specific subject performing a rare action on an object).
- Spatial Relations: Joy Hsu et al. show in NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations that separating the representation of intrinsic object attributes from extrinsic spatial relations prevents models from relying on dataset-specific spatial biases, improving generalization in novel environments.
Decoupling Dynamics and Interaction Rules
Compositional generalization is equally vital for understanding physical laws and dynamics. If a model entangles an object's visual appearance with the rules governing its behavior, it cannot predict how a newly encountered object will interact with its environment.
To decouple "what" an object is from "how" it behaves, Aniket Didolkar et al. introduce Neural Production Systems. By separating the definition of an interaction rule from the specific properties of the entities involved, their architecture can apply a learned behavior—such as a collision rule—to novel object types without additional training.
Holistic Physics Model Compositional Law Parsing
┌──────────────────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Object Appearance + Dynamics │ │ Object Latents │ │ Latent Physics │
│ (Overfits to training shapes)│ │ (Shape, Texture) │ │ (Collision, Gravity)│
└──────────────────────────────┘ └────────┬─────────┘ └────────┬─────────┘
└───────────┬────────────┘
▼
Zero-Shot Dynamics Prediction
on Unseen Object Appearances
Similarly, the authors of Compositional Law Parsing with Latent Random Functions separate static object attributes from the latent functions governing their motion. Because the learned rules are independent of appearance features, the model can accurately predict the dynamics of objects with entirely unseen visual properties. This form of operational factorization is also highly effective for learning environmental transformations; Barthélémy Dang-Nhu et al. demonstrate in Disentangled representation learning through unsupervised symmetry group discovery that discovering independent symmetry groups for actions allows an agent to correctly predict the effects of transformations in completely novel configurations.
Training Strategies for Inducing Disentanglement
How do we force a neural network to learn these factorized representations without relying on exhaustive manual annotations? Researchers have developed several distinct training paradigms to incentivize independence:
Unsupervised and Structured Generative Modeling
One foundational approach is to embed statistical constraints directly into generative architectures. Xander Steenbrugge et al. demonstrate in Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations that using an unsupervised Variational Autoencoder (VAE) to pre-learn independent generative factors (like shape and position) provides a robust foundation for solving abstract reasoning tasks. To isolate interactions cleanly, Kaspar Märtens et al. introduce Neural Decomposition: Functional ANOVA with Variational Autoencoders, which embeds a functional ANOVA structure into a VAE. By isolating specific marginal effects from interaction terms, the model can accurately synthesize data points representing attribute pairings that were entirely absent during training.
Weak Supervision and Environmental Robustness
When full unsupervised separation is difficult, weak supervision can guide latent alignment. Yordan Hristov et al. show in Interpretable Latent Spaces for Learning from Demonstration that using weak supervision to untangle latent dimensions along human concepts allows robots to ground abstract instructions into real-world objects and recognize novel combinations. Furthermore, this factorization provides environmental resilience; Nick Pawlowski et al. demonstrate in Representation Disentanglement for Multi-task Learning with application to Fetal Ultrasound that explicitly separating core anatomical features from environmental artifacts allows models to maintain high accuracy when familiar features appear in novel, out-of-distribution contexts.
Predictive Objectives and Compositional Sampling
Disentanglement can also emerge naturally from predictive tasks or structured data sampling:
- Predictive World Models: In Probing Emergent Semantics in Predictive Agents via Question Answering, Abhishek Das et al. discover that agents trained purely to predict future sensory observations spontaneously develop factorized internal representations of color, shape, and position, unlocking emergent compositional reasoning without explicit labels.
- Semi-Positive Compositional Sampling: To explicitly prevent a model from memorizing fixed triplets during training, Fang Nan et al. propose a training strategy in Visual-guided human-object interaction detection using semi-positive examples. By sourcing different components of a target relationship (such as the action and the object) from separate images, the model is forced to learn independent representations for each primitive, enabling it to recognize novel combinations it has never seen as a single unit.
Generalization Beyond Vision
While visual attribute composition is an intuitive sandbox, factorized representations are equally powerful in non-visual domains.
In natural language processing, traditional sentiment models often struggle when multiple entities and aspects are discussed simultaneously. To resolve this, Jun Yang et al. develop the CEA method in Multi-Entity Aspect-Based Sentiment Analysis With Context, Entity and Aspect Memory. By learning independent semantic representations for entities and aspects before combining them, the model successfully predicts sentiment for specific pairs it never encountered during training, bypassing memorized co-occurrence patterns.
In spatiotemporal forecasting, models often fail when data is fragmented or when environmental conditions shift unexpectedly. Zhengyang Zhou et al. address this in Towards Learning in Grey Spatiotemporal Systems: A Prophet to Non-consecutive Spatiotemporal Dynamics by decoupling environmental factors from historical spatial observations. Because these predictive influences are separated into distinct components, the model generalizes seamlessly to novel combinations of conditions (such as an unobserved weather pattern occurring on a specific day of the week), ensuring robust predictions even when recent data is missing.
Why It Works: Beating the Combinatorial Explosion
Ultimately, factorized representation learning works because it fundamentally alters how a model scales with data complexity.
Consider a domain with $N$ attributes and $M$ objects. A monolithic model that attempts to learn the joint distribution $P(\text{attribute}, \text{object})$ must see a significant fraction of the $N \times M$ combinations to generalize reliably. As the number of primitives grows, the data requirement suffers from an exponential combinatorial explosion.
Factorization breaks this curse of dimensionality. By constraining the architecture to learn independent distributions—$P(\text{attribute})$ and $P(\text{object})$—and combining them via a deterministic, structural operator, the model only needs to see each primitive a linear number of times. Generalization to an unseen combination ceases to be an extrapolation problem; instead, it becomes a simple, zero-shot operation of re-composing independent coordinates within a structured latent space.
Key Foundational Work
Multi-Entity Aspect-Based Sentiment Analysis With Context, Entity and Aspect Memory (2018) — contributed 2 concept elements to this direction.
Interpretable Latent Spaces for Learning from Demonstration (2018) — contributed 1 concept elements to this direction.
Learning Compositional Sparse Models of Bimodal Percepts (2014) — contributed 1 concept elements to this direction.