The Failure of Scale for Compositional Reasoning

This direction is merged into another direction: compositional generalization (compositional generalization gap). Its last version stays readable and citable here.

Version 1 of 124 papers34 conceptsfrozen 2026-10-04

In the current era of deep learning, the prevailing wisdom is that "scale is all you need." We have seen that as models grow in parameter count and training data, they become remarkably better at translation, coding, and conversation. However, a growing body of research suggests that one fundamental human ability remains stubbornly resistant to scaling: compositional generalization.

Compositionality is the ability to understand and produce a potentially infinite number of new combinations from a finite set of known components. If you know what a "red cube" is and what a "blue sphere" is, you should immediately understand a "red sphere," even if you have never seen one. While humans do this effortlessly, modern neural networks—from Large Language Models (LLMs) to Vision-Language Models (VLMs)—frequently fail this "Lego-like" test.

The Scaling Myth

The most striking finding across recent studies is that increasing the size of a model does not inherently fix its compositional weaknesses. In *-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task, Dmitry Tsarkov and colleagues demonstrate that the performance gap between random data splits and compositional splits remains massive even as training data scales.

This sentiment is echoed by Kate McCurdy and Paul Smolensky in their survey, Toward Compositional Behavior in Neural Models: A Survey of Current Views. They find a strong consensus among researchers that simply "scaling up" is insufficient for achieving human-like systematic generalization. Whether it is Sanket Vaibhav Mehta looking at data-to-text generation or Keito Kudo testing arithmetic reasoning, the conclusion is the same: larger models might achieve higher absolute accuracy, but they suffer from the same relative performance drop-off when faced with novel combinations of known rules.

The Systematicity Gap Across Domains

The failure of compositionality isn't limited to text; it spans every modality where neural networks are applied.

  • Vision and Language: In the CREPE benchmark, Zixian Ma and Irena Gao show that foundation models like CLIP struggle to understand how objects, attributes, and relations combine. A model might recognize a "dog" and a "car," but fail to distinguish "a dog in a car" from "a car in a dog."
  • Mathematical Proofs: Haoyu Zhao introduced Ineq-Comp, revealing that LLMs that solve a base math problem often fail completely when that problem is slightly modified or combined with another, suggesting they lack a robust internal logic for composing proofs.
  • Physical World Models: Even breathtaking video generation models like Sora appear to rely on "case-based mimicry." Bingyi Kang and Yang Yue found that these models excel at interpolating within their training data but fail to learn universal physical laws, prioritizing visual features like color over fundamental properties like shape in How Far Is Video Generation from World Model: A Physical Law Perspective.

Why Models Struggle: Memorization vs. Rules

Why does scale fail here? The evidence suggests that neural models are biased toward lexical generalization—swapping words into known patterns—rather than structural generalization.

Alexander Koller and Yuekun Yao argue in Structural generalization is hard for sequence-to-sequence models that the bottleneck is often the decoder. While the encoder may capture structural information, the decoder fails to utilize it to generate novel structures, preferring shallow patterns seen during training.

Furthermore, models learn what is supposed to be together based on co-occurrence statistics. Helen Qu and Sang Michael Xie demonstrate that multimodal models fail on rare concept combinations because they over-rely on pre-training word co-occurrences. They aren't learning the "rule" of what an object is; they are learning the "vibe" of which objects usually appear together.

Beyond Raw Scale: Diversity and Structure

If scale isn't the answer, what is? Several researchers point toward the composition of the data rather than its volume.

Ian Berlot-Attwell shows in Attribute Diversity Determines the Systematicity Gap in VQA that training on a higher diversity of attribute combinations, rather than just more data, is what helps models learn underlying rules. Similarly, Xiang Zhou and Mohit Bansal found that high dataset complexity and low repetition force models to abandon rote memorization in favor of compositional rules.

Other promising directions include Iterated Learning, where models learn from their own generated outputs to refine their understanding of parts and wholes (Chenhao Zheng), and specialized pre-training. For instance, pre-training on code has been shown to act as a "secret sauce" for better structural reasoning in language models, likely because code is inherently hierarchical and compositional.

Ultimately, the research suggests that compositionality is an algorithmic and structural challenge. To bridge the gap, we must move beyond the brute force of scaling and focus on how models represent the relationship between the parts and the whole.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_failure_of_scale_for_compositional_r_1db992c0,
  title = {The Failure of Scale for Compositional Reasoning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-failure-of-scale-for-compositional-reasoning/rdr_1db992c094a64e7aa338b3334cf7c915},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_failure_of_scale_for_compositional_r_1db992c0_v1,
  title = {The Failure of Scale for Compositional Reasoning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-failure-of-scale-for-compositional-reasoning/rdr_1db992c094a64e7aa338b3334cf7c915/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}