COGITAO: A Visual Reasoning Framework To Study Compositionality & Generalization

Yassine Taoudi-Benchekroun, Klim Troyan, Pascal Sager, Stefan Gerber, Lukas Tuggener, Benjamin F. GreweOpen 2025 · DOI · arXiv

Introduction

While modern machine learning models demonstrate remarkable proficiency in pattern matching and statistical correlation, they frequently struggle with compositional generalization—the ability to combine known concepts in novel ways. This capacity is a fundamental pillar of human intelligence, allowing us to understand and execute complex instructions like "rotate the red object twice" even if we have never encountered that specific sequence before.

In the visual domain, existing benchmarks often conflate reasoning challenges with perceptual complexity or suffer from limited scalability. To address this, we introduce COGITAO, a highly controllable, procedural grid-world generator designed to isolate and probe compositional generalization in visual AI. By providing a clean, abstract environment, COGITAO allows researchers to diagnose whether a model's failure stems from visual noise or a fundamental inability to perform relational reasoning.

The COGITAO Framework

COGITAO is designed as a procedural data-generation engine that creates rule-based visual tasks on 2D grids. The framework is built upon three core components:

  1. Atomic Objects: A pool of 23,000 pre-generated objects varying in shape, size, symmetry, and color.
  2. Transformations: A library of 28 atomic object-transformations, including translation, rotation, mirroring, color changes, and padding.
  3. Procedural Rules: The generator strings these transformations together at adjustable depths to create complex rules (e.g., "translate up, then rotate 90 degrees").

This modularity allows for the creation of virtually infinite task variations, enabling precise control over the distribution of training and testing data.

Experimental Methodology

To evaluate current architectural paradigms, we established two primary evaluation tracks:

  • CompGen (Compositional Generalization): Models are trained on atomic rules and specific combinations (depth-1 and depth-2). They are then tested on unseen combinations of those same rules to measure systematicity.
  • EnvGen (Environmental Generalization): Models are trained on fixed rules but tested on out-of-distribution (OOD) environments, such as larger grids or a higher number of objects than seen during training.

Baseline Architectures

We benchmarked four distinct architectures, each approximately 1.2M parameters, trained from scratch:

  • Vanilla Vision Transformer (ViT): A standard attention-based model.
  • Grid-TF: A transformer variant infused with grid-specific spatial biases and object positional encodings.
  • Pondering Looped TF (PL-TF): A model utilizing recurrence and adaptive compute time to mimic iterative, multi-step reasoning.
  • LLaDA: A diffusion-based language model adapted for grid representation.

Key Insights and Results

The Generalization Collapse

Our experiments reveal a stark contrast between in-distribution performance and out-of-distribution generalization. While all models achieved high accuracy (90–99%) on training distributions, they suffered a catastrophic collapse on CompGen OOD tasks. Specifically, Vanilla-TF, Grid-TF, and LLaDA scored 0.0% accuracy on unseen rule combinations. The PL-TF model was the only architecture to solve any OOD tasks, though its performance remained marginal at 0.1%.

The "Stubbornness" Phenomenon

Analysis of model errors reveals a specific failure mode termed "stubbornness." Rather than producing random noise when faced with novel rule combinations, models frequently output a grid that perfectly matches a different rule combination encountered during training. This suggests that models are overfitting to the specific compositional structures of the training set rather than learning the underlying atomic transformations.

Scaling Limits

To test if scale could mitigate these failures, we increased the training data to 1 million samples and scaled model parameters up to 25M. Despite these increases, the generalization gap remained unchanged. Furthermore, testing frontier Large Language Models (LLMs) like GPT-4o and Gemini 1.5 Pro via In-Context Learning yielded the same precipitous drop in performance on OOD tasks, confirming that massive scale alone does not inherently produce systematic generalization.

The Promise of Recurrence

The PL-TF model's slight edge in robustness suggests that inductive biases toward iterative processing may be more aligned with compositional tasks than standard feed-forward attention. Weight-sharing through looping allows the model to apply transformations sequentially, mimicking the multi-step nature of the underlying rules.

Conclusion and Future Work

COGITAO exposes a fundamental structural limitation in current AI architectures: the inability to decompose and recompose visual rules systematically. The failure of both specialized small-scale models and massive frontier LLMs highlights the need for new inductive biases that prioritize object-centric and transformation-centric abstractions.

Future research using COGITAO will focus on curriculum learning strategies and internal representation analysis. By understanding how (or if) models form abstract concepts of "objects" and "actions," we can move toward designing architectures capable of true compositional intelligence.

Summary updated

Abstract

The ability to compose learned concepts and apply them in novel settings is key to human intelligence, but remains a persistent limitation in state-of-the-art machine learning models. To address this issue, we introduce COGITAO, a modular and extensible data generation framework and benchmark designed to systematically study compositionality and generalization in visual domains. Drawing inspiration from ARC-AGI's problem-setting, COGITAO constructs rule-based tasks which apply a set of transformations to objects in grid-like environments. It supports composition, at adjustable depth, over a set of 28 interoperable transformations, along with extensive control over grid parametrization and object properties. This flexibility enables the creation of millions of unique task rules -- surpassing concurrent datasets by several orders of magnitude -- across a wide range of difficulties, while allowing virtually unlimited sample generation per rule. We provide baseline experiments using state-of-the-art vision models, highlighting their consistent failures to generalize to novel combinations of familiar elements, despite strong in-domain performance. COGITAO is fully open-sourced, including all code and datasets, to support continued research in this field.
BibTeX
@misc{taoudibenchekroun2025cogitao,
  author = {Yassine Taoudi-Benchekroun and Klim Troyan and Pascal Sager and Stefan Gerber and Lukas Tuggener and Benjamin F. Grewe},
  title = {{COGITAO: A Visual Reasoning Framework To Study Compositionality \& Generalization}},
  year = {2025},
  eprint = {2509.05249},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.05249},
}
Figure