When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks

Ankur Sikarwar, Arkil Patel, Navin Goyal

Abstract

Humans possess the innate ability to reason compositionally while grounding language to the physical world. Recent benchmarks like ReaSCAN use grid-world navigation tasks to assess whether neural models exhibit similar capabilities. In this work, we present GroCoT (Grounded Compositional Transformer), a simple transformer-based model that outperforms specialized architectures on ReaSCAN and gSCAN. Our analysis reveals that identifying the target location in the grid world is the primary challenge for these models, rather than navigation logic. Furthermore, we demonstrate that the "C2" depth generalization split in ReaSCAN is flawed; on an amended "fair" version, transformers generalize effectively to deeper input structures. Finally, we introduce RefEx, a simplified grounded task, to derive a precise mathematical construction of how a single self-attention head performs compositional grounding through attribute summation.

Introduction

Compositional generalization—the ability to understand novel combinations of known components—is a cornerstone of human intelligence. In grounded language learning, this requires mapping linguistic utterances (e.g., "the small red square") to specific entities in a visual environment. While humans perform this effortlessly, standard neural architectures often struggle, leading researchers to develop complex, task-specific models.

This paper investigates whether standard Transformers are truly limited in these settings or if their perceived failures stem from suboptimal world representations and evaluation metrics. By introducing GroCoT and the RefEx benchmark, we demonstrate that Transformers are highly capable of grounded compositional reasoning when properly configured.

The GroCoT Architecture

To address the limitations of previous models, we developed the Grounded Compositional Transformer (GroCoT). GroCoT introduces three key architectural refinements to the standard multimodal Transformer:

  1. Improved Spatial Representation: Traditional models often use CNNs with limited receptive fields to process grid worlds. GroCoT instead tokenizes each grid cell, projecting them into a high-dimensional space with 2D positional encodings to preserve spatial relationships.
  2. Interleaved Self-Attention: Rather than immediate fusion, GroCoT interleaves self-attention layers (processing language and vision independently) with cross-modal attention layers. This allows each modality to develop robust internal representations before synchronization.
  3. Disambiguated Encoding: To prevent attribute "blurring"—a common issue when multiple objects occupy the same space (e.g., a cylinder inside a box)—we modified the object representation to ensure that overlapping attributes remain distinct and recoverable.

Benchmarking and Datasets

We evaluated GroCoT across several established benchmarks and one novel task:

  • gSCAN & GSRR: Navigation tasks requiring models to follow commands like "walk to a small red circle" in a 2D grid.
  • ReaSCAN: A complex extension of gSCAN involving relative clauses and recursive relationships (e.g., "the box in the same row as the green square").
  • RefEx (Referring Expressions): A new, simplified benchmark designed to isolate target identification from navigation. This allows for a granular analysis of how Transformers ground and compose specific attributes.

Key Insights and Results

The Target Identification Bottleneck

Through linear probing, we discovered that model failures in navigation tasks are rarely due to pathfinding errors. Instead, the primary bottleneck is target identification. When GroCoT is provided with the "gold" (correct) target location, it achieves near 100% navigation accuracy across all splits. This suggests that the core difficulty of grounded compositional generalization lies in the linguistic-visual mapping process.

Rethinking Depth Generalization

We identified a flaw in the ReaSCAN "C2" test split, which evaluates depth generalization. The original training data lacked sufficient information for models to distinguish between different recursive clause types. By creating a "fair" version of this split (C2-deeper), we demonstrated that Transformers can indeed generalize to deeper structures than those seen during training, provided the training distribution is representative.

Transformers as Attribute Summation Engines

Using the RefEx task, we performed a mathematical analysis of the model's weights. We proved that a single self-attention head can identify a target by effectively performing a logical "AND" operation. By "summing up" matches for individual attributes (e.g., "red" + "square"), the attention mechanism assigns the highest scores to the correct grid cells. This construction allows for 100% accuracy on novel attribute combinations in a zero-shot setting.

Performance Summary

GroCoT significantly outperformed previous state-of-the-art specialized models:

  • GSRR: Increased accuracy from 94.7% to 99.9%.
  • ReaSCAN: Increased accuracy from 69.9% to 82.2%.

Limitations and Future Work

While GroCoT excels at target identification, it still faces challenges with sequence-side generalization—specifically gSCAN splits that require novel sequences of actions (e.g., performing a new movement pattern). Additionally, our current mathematical interpretations are focused on the simplified RefEx task; extending this to the recursive complexities of ReaSCAN remains a priority. Finally, while grid worlds provide a controlled environment for study, grounding language in high-dimensional, "entangled" real-world imagery is the next major frontier.

Conclusion

Our findings suggest that standard Transformer architectures are more capable of grounded compositional reasoning than previously thought. By focusing on spatial representation and modality-specific processing, GroCoT sets a new standard for these benchmarks. Furthermore, our mathematical analysis provides a clear roadmap for understanding how these models compose attributes, moving us closer to interpretable grounded AI.

Summary updated

Abstract

Humans can reason compositionally whilst grounding language utterances to the real world. Recent benchmarks like ReaSCAN use navigation tasks grounded in a grid world to assess whether neural models exhibit similar capabilities. In this work, we present a simple transformer-based model that outperforms specialized architectures on ReaSCAN and a modified version of gSCAN. On analyzing the task, we find that identifying the target location in the grid world is the main challenge for the models. Furthermore, we show that a particular split in ReaSCAN, which tests depth generalization, is unfair. On an amended version of this split, we show that transformers can generalize to deeper input structures. Finally, we design a simpler grounded compositional generalization task, RefEx, to investigate how transformers reason compositionally. We show that a single self-attention layer with a single head generalizes to novel combinations of object attributes. Moreover, we derive a precise mathematical construction of the transformer's computations from the learned network. Overall, we provide valuable insights about the grounded compositional generalization task and the behaviour of transformers on it, which would be useful for researchers working in this area.
BibTeX
@inproceedings{sikarwar2022when,
  author = {Ankur Sikarwar and Arkil Patel and Navin Goyal},
  title = {{When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=2pRp19SM8t1},
}
Figure