Modular and Symbolic Mechanisms for Compositional Generalization

16 concepts16 papers1 proposal

If a deep learning model learns to recognize a "red suitcase" and a "blue car" during training, it should theoretically have no trouble recognizing a "blue suitcase" at test time. To a human, this transition is trivial because we naturally decompose our world into independent, reusable primitives like colors and objects. However, standard neural networks frequently fail this test. Instead of learning isolated concepts, they entangle features, memorizing the specific combinations present in their training data. When faced with an unfamiliar combination of familiar attributes, their performance drops precipitously—a phenomenon known as the compositional generalization gap.

To bridge this gap, researchers are moving away from monolithic architectures. Instead, they are designing systems that explicitly isolate, route, and recombine concepts using modular networks, structured graphs, and symbolic inference engines.

The Modularity Paradox: Routing and Coupling

The most intuitive way to enforce compositionality is to build it directly into the network's architecture. If a model consists of separate, specialized neural modules—such as one module for detecting shapes and another for identifying colors—it can dynamically assemble these pieces to process novel inputs.

For instance, Boris Katz et al. demonstrated this by parsing linguistic commands into structural trees and assembling corresponding neural modules on the fly, forcing them to communicate via shared attention maps. To make this process scalable and adaptive, Minyang Hu et al. introduced an "agreement router" that uses iterative message passing to dynamically select and compose specialist modules based on the input task.

Standard Monolithic Network:
Input ("Blue Suitcase") ──> [ Entangled Representation ] ──> Failure (OOD)

Modular Architecture with Routing:
Input ("Blue Suitcase") ──> [ Router ] ──> [ Blue Module ] ──┐
                                       ──> [ Suitcase Module ] ─┴─> Success

However, simply dividing a network into modules does not guarantee they will learn independent concepts. In a rigorous theoretical analysis, Devon Jarvis et al. revealed a fundamental mathematical hurdle: during gradient descent, neural modules naturally "couple" with one another unless the architecture perfectly segregates task structures. When modules couple, they co-adapt to the training distribution, destroying their ability to generalize to novel combinations.

To counteract this, the degree of modularity must be carefully calibrated. Vanessa D'Amario et al. discovered a "Goldilocks zone" of intermediate modularity, showing that grouping related sub-tasks together—particularly at the early image encoding stage—prevents destructive coupling and maximizes out-of-distribution generalization.

Grounding Concepts via Dictionaries and Graphs

Another strategy to prevent feature entanglement is to anchor neural representations to explicit conceptual targets. Without an external reference point, a vision model's internal representation of an object can easily drift depending on its surrounding context.

To anchor these representations, Xiaojian Ma et al. introduced a "concept-feature dictionary" into Vision Transformers. By using auxiliary tasks that align current visual features with a persistent memory bank of previously seen concepts, the model preserves the identity of individual attributes regardless of how they are paired. Similarly, Arjun Akula et al. utilized language-guided adaptive convolutions to condition visual filters directly on textual queries, forcing the visual system to isolate the specific relations demanded by the text.

Structuring the data itself as a graph can also preserve these boundaries. Raeid Saqur et al. mapped both images and text into fine-grained relational graphs, using a cross-modal correspondence matrix to ensure that individual objects and relations maintain their distinct identities. This structural organization can even emerge without supervision: Zhijin Guo et al. showed that modeling polyphonic music as a multi-relational graph naturally forces unsupervised embeddings to organize themselves into independent, additive factors like composer, genre, and voice.

Symbolic Verification and Test-Time Inference

When continuous neural representations are insufficient to maintain strict compositional boundaries, researchers often turn to hybrid systems that combine neural pattern recognition with symbolic logic.

One prominent approach is the dual-process framework. Alexander Noviello et al. developed an architecture where a fast, intuitive neural network ("System 1") generates candidate interpretations, which are then strictly verified and filtered by a rule-based symbolic engine ("System 2"). To make this interaction differentiable, Shanmukha Ramakrishna Vedantam et al. treated symbolic reasoning programs as latent, stochastic variables within a probabilistic framework, optimizing them via variational inference. This allows the model to dynamically "debug" its own logical execution paths when encountering novel scenarios.

Dual-Process Inference:
Input ──> [ System 1: Neural Network ] ──> Candidate Hypotheses ──┐
                                                                 ▼
                                              [ System 2: Symbolic Engine ]
                                                                 │
                                                                 ▼
                                                          Verified Output

Furthermore, static weights learned during training are often inherently biased toward the training distribution. To overcome this, Haruki Shirakami et al. introduced the Abduction Transformer, which shifts the burden of generalization from training to inference. By representing knowledge as a nonparametric mixture of distributions, the model performs gradient-based search at test time to dynamically fine-tune its hypothesis for entirely new types of puzzles.

Data Symmetries and Discrete Communication

While architectural modifications are powerful, compositionality can also be framed as a mathematical constraint on the data distribution itself. Rather than changing the model, can we force standard networks to generalize by altering how they interact with data?

Ekin Akyürek et al. explored this through LexSym, a model-agnostic data augmentation tool that automatically discovers and enforces "lexical symmetries." By systematically swapping invariant properties in the training data, it explicitly penalizes the network for memorizing specific attribute combinations.

This data-centric view aligns with a broader insight regarding how models communicate internally. Marc Niethammer et al. investigated whether continuous "disentangled" latent spaces (like those in VAEs) actually lead to systematic generalization. Surprisingly, they found that standard continuous disentanglement metrics do not predict generalization performance. Instead, models that communicate via discrete symbols—resembling an emergent language—consistently outperform continuous models. Discrete symbols act as a natural bottleneck, preventing the bleeding of features and forcing the system to operate on clean, compositional primitives.

Key Foundational Work

Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering (2019) — contributed 1 concept elements to this direction.
SCAN: Learning Hierarchical Compositional Visual Concepts (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_systematic_generalization_95a4e2d6,
  title = {systematic generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/systematic-generalization/rdr_95a4e2d62ca2473e88936a4c4df90d9b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_systematic_generalization_95a4e2d6_v1,
  title = {systematic generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/systematic-generalization/rdr_95a4e2d62ca2473e88936a4c4df90d9b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake