Introduction
While modern machine learning models demonstrate remarkable proficiency in pattern matching and statistical correlation, they frequently struggle with compositional generalization—the ability to combine known concepts in novel ways. This capacity is a fundamental pillar of human intelligence, allowing us to understand and execute complex instructions like "rotate the red object twice" even if we have never encountered that specific sequence before.
In the visual domain, existing benchmarks often conflate reasoning challenges with perceptual complexity or suffer from limited scalability. To address this, we introduce COGITAO, a highly controllable, procedural grid-world generator designed to isolate and probe compositional generalization in visual AI. By providing a clean, abstract environment, COGITAO allows researchers to diagnose whether a model's failure stems from visual noise or a fundamental inability to perform relational reasoning.
The COGITAO Framework
COGITAO is designed as a procedural data-generation engine that creates rule-based visual tasks on 2D grids. The framework is built upon three core components:
- Atomic Objects: A pool of 23,000 pre-generated objects varying in shape, size, symmetry, and color.
- Transformations: A library of 28 atomic object-transformations, including translation, rotation, mirroring, color changes, and padding.
- Procedural Rules: The generator strings these transformations together at adjustable depths to create complex rules (e.g., "translate up, then rotate 90 degrees").
This modularity allows for the creation of virtually infinite task variations, enabling precise control over the distribution of training and testing data.
Experimental Methodology
To evaluate current architectural paradigms, we established two primary evaluation tracks:
- CompGen (Compositional Generalization): Models are trained on atomic rules and specific combinations (depth-1 and depth-2). They are then tested on unseen combinations of those same rules to measure systematicity.
- EnvGen (Environmental Generalization): Models are trained on fixed rules but tested on out-of-distribution (OOD) environments, such as larger grids or a higher number of objects than seen during training.
Baseline Architectures
We benchmarked four distinct architectures, each approximately 1.2M parameters, trained from scratch:
- Vanilla Vision Transformer (ViT): A standard attention-based model.
- Grid-TF: A transformer variant infused with grid-specific spatial biases and object positional encodings.
- Pondering Looped TF (PL-TF): A model utilizing recurrence and adaptive compute time to mimic iterative, multi-step reasoning.
- LLaDA: A diffusion-based language model adapted for grid representation.
Key Insights and Results
The Generalization Collapse
Our experiments reveal a stark contrast between in-distribution performance and out-of-distribution generalization. While all models achieved high accuracy (90–99%) on training distributions, they suffered a catastrophic collapse on CompGen OOD tasks. Specifically, Vanilla-TF, Grid-TF, and LLaDA scored 0.0% accuracy on unseen rule combinations. The PL-TF model was the only architecture to solve any OOD tasks, though its performance remained marginal at 0.1%.
The "Stubbornness" Phenomenon
Analysis of model errors reveals a specific failure mode termed "stubbornness." Rather than producing random noise when faced with novel rule combinations, models frequently output a grid that perfectly matches a different rule combination encountered during training. This suggests that models are overfitting to the specific compositional structures of the training set rather than learning the underlying atomic transformations.
Scaling Limits
To test if scale could mitigate these failures, we increased the training data to 1 million samples and scaled model parameters up to 25M. Despite these increases, the generalization gap remained unchanged. Furthermore, testing frontier Large Language Models (LLMs) like GPT-4o and Gemini 1.5 Pro via In-Context Learning yielded the same precipitous drop in performance on OOD tasks, confirming that massive scale alone does not inherently produce systematic generalization.
The Promise of Recurrence
The PL-TF model's slight edge in robustness suggests that inductive biases toward iterative processing may be more aligned with compositional tasks than standard feed-forward attention. Weight-sharing through looping allows the model to apply transformations sequentially, mimicking the multi-step nature of the underlying rules.
Conclusion and Future Work
COGITAO exposes a fundamental structural limitation in current AI architectures: the inability to decompose and recompose visual rules systematically. The failure of both specialized small-scale models and massive frontier LLMs highlights the need for new inductive biases that prioritize object-centric and transformation-centric abstractions.
Future research using COGITAO will focus on curriculum learning strategies and internal representation analysis. By understanding how (or if) models form abstract concepts of "objects" and "actions," we can move toward designing architectures capable of true compositional intelligence.
Summary updated