Testing Compositional Generalization through Systematic Data Splits

114 concepts94 papers2 proposals

Deep learning models often appear remarkably intelligent until they are tested on something slightly outside their training comfort zone. If a model learns what a "red sphere" and a "blue cube" look like, can it immediately recognize a "red cube"?

In standard machine learning evaluations, datasets are split randomly into training and testing sets. Because both sets are drawn from the same underlying distribution (i.e., independent and identically distributed, or i.i.d.), the test set inevitably contains many of the same attribute combinations seen during training. As shown by Justin Johnson et al. in CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning, this allows models to rely on superficial statistical shortcuts and linguistic biases rather than developing a true, modular understanding of the world.

To expose these shortcuts, a prominent research direction focuses on compositional generalization testing. The core idea is to force models to evaluate novel combinations of familiar building blocks by systematically engineering the data splits.

The Anatomy of a Compositional Split: Atoms vs. Compounds

To build intuition for how these splits work, we can look at the Distribution-Based Compositionality Assessment (DBCA) framework formalized by Xiao Wang in Measuring Compositional Generalization: A Comprehensive Method on Realistic Data. Wang breaks data down into two distinct levels:

  1. Atoms: The primitive, indivisible units of meaning or features (e.g., the color red, the shape cube, the verb throw, the noun ball).
  2. Compounds: The combinations or structural arrangements of those atoms (e.g., red cube, a child throws a ball).

In a standard random split, both the atom distribution and the compound distribution remain identical between the training and testing sets.

In a compositional split, we intentionally manipulate these distributions. We ensure that the training and testing sets share the exact same vocabulary of atoms—giving the model an equal opportunity to learn what each primitive means—but we maximize the divergence between their compounds. The test set consists entirely of novel combinations that were explicitly withheld from training.

Standard i.i.d. Split:
Train: [Red Sphere], [Blue Cube], [Red Cube]
Test:  [Red Sphere], [Blue Cube]  <-- Memorization works!

Compositional Split:
Train: [Red Sphere], [Blue Cube]
Test:  [Red Cube],   [Blue Sphere] <-- Requires true compositionality!

Algorithmic Partitioning: How to Split the Space

Creating a clean separation of compounds is computationally challenging. If you randomly drop samples, you will accidentally include overlapping combinations. Researchers have developed several algorithmic strategies to partition data effectively:

Dimensions of Compositionality

Compositional generalization manifests across several distinct axes, each testing a different facet of systematic reasoning:

Attribute Combinations and Disentanglement

This dimension tests if a model can recognize known features in entirely unseen configurations. For example, Vignesh Ramanathan et al. developed PACO: Parts and Attributes of Common Objects to measure a model's capacity to identify objects based on novel combinations of previously learned parts and attributes (such as a "blue handle" on a "white mug").

Structural and Grammatical Roles

This dimension involves moving a known primitive to a completely new structural position. Najoung Kim et al. designed COGS: A Compositional Generalization Challenge Based on Semantic Interpretation to test if a model can understand a noun when it transitions from the subject position during training to the object position during testing.

Productivity and Length Scaling

This dimension evaluates whether a model can chain known operations into longer, more complex sequences than those seen during training. As demonstrated by Brenden Lake et al. in Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks, standard sequence-to-sequence models often fail spectacularly when asked to apply a newly learned verb to longer structural modifiers.

Universality of the Paradigm

While systematic data partitioning originated largely in visual question answering and natural language processing, it has expanded into a universal diagnostic tool across diverse scientific domains:

By shifting the evaluation paradigm from random splits to systematic, disjoint compound splits, this research direction provides the mathematical friction necessary to diagnose whether an AI model has truly mastered the underlying rules of a domain or is merely a sophisticated pattern matcher.

Key Foundational Work

C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset (2017) — contributed 2 concept elements to this direction.
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (2017) — contributed 1 concept elements to this direction.
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_generalization_testing_a714cced,
  title = {compositional generalization testing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-testing/rdr_a714cced98a147f58564b97aa0eac64a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_compositional_generalization_testing_a714cced_v1,
  title = {compositional generalization testing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-generalization-testing/rdr_a714cced98a147f58564b97aa0eac64a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake