When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks

Ankur Sikarwar, Arkil Patel, Navin Goyal

Introduction

Compositional generalization is a hallmark of human intelligence—the ability to understand novel combinations of known components. For instance, if a person understands the concept of a "red cube" and the action of "spinning," they can immediately interpret the command to "spin the red cube," even if they have never witnessed that specific combination before.

In the domain of grounded language understanding, artificial intelligence models must map natural language commands to specific actions within a visual environment. While humans perform these tasks intuitively, neural models frequently struggle with "out-of-distribution" (OOD) scenarios where the logic or attribute combinations differ from the training data. This paper investigates whether generic Transformer architectures can overcome these hurdles and identifies the specific bottlenecks preventing generalization.

Figure 1: An example of a grounded language task where an agent must interpret a complex command to perform a sequence of actions in a grid world.

The GroCoT Architecture

The authors introduce GroCoT (Grounded Compositional Transformer), a model that applies three targeted enhancements to the standard multi-modal Transformer to improve its reasoning capabilities:

  1. Improved Spatial Representation: Standard CNNs often have limited receptive fields. GroCoT tokenizes each grid cell individually and incorporates 2D positional encodings, allowing the model to reason about spatial relationships across any distance.
  2. Interleaving Self-Attention: Instead of immediately mixing vision and language through cross-modal attention, GroCoT interleaves self-attention layers. This allows each modality to develop robust internal representations before synchronization.
  3. Modified World State Encoding: The model addresses a common flaw in representing overlapping objects (e.g., a cylinder inside a box). By ensuring the encoding can disambiguate the attributes of a container from its contents, the model avoids confusion during complex spatial reasoning.

Benchmarking Compositional Logic

The study evaluates GroCoT across several established and novel benchmarks designed to test different facets of compositional logic. These include gSCAN and GSRR for navigation, and ReaSCAN, which introduces relative clauses and complex spatial relations.

Figure 4: Examples of different task complexities in benchmarks, ranging from simple attribute matching (Two-Attr) to complex relational expressions (Three-Attr-Rel).

To further probe the model's limits, the authors introduced C2-deeper, a split designed to test generalization to deeper command structures (e.g., multiple nested "that is" clauses). They also created RefEx, a simplified "Referring Expressions" task used to isolate target identification from action execution for mathematical analysis.

Key Insights and Results

Target Identification is the Primary Bottleneck

Through "gold target" experiments—where the model is explicitly told the target's location—the authors found that navigation accuracy jumps to nearly 100%. This proves that the failure of current models lies almost entirely in identifying which object the command refers to, rather than the subsequent planning of actions.

State-of-the-Art Performance

GroCoT significantly outperformed previous specialized architectures. On the GSRR benchmark, it achieved 98.8% accuracy, a massive leap from the previous best of 63.5%. On the complex ReaSCAN benchmark, it reached 82.2% accuracy. Furthermore, on the C2-deeper split, GroCoT achieved 85.6%, demonstrating that Transformers can handle recursive logic if the training distribution is fair.

Mathematical Transparency

Using the RefEx task, the authors provided a rare look into the "black box" of Transformer logic. They proved that a single-layer, single-head attention mechanism is sufficient to ground attributes. By deriving a mathematical construction from learned weights, they showed the model performs a series of dot products that effectively act as "attribute filters" to isolate the target object.

Conclusion and Future Work

While GroCoT excels at target identification and recursive logic, challenges remain. The model still struggles with certain gSCAN splits that require generalizing to entirely novel output sequences (e.g., moving in a direction never seen during training). Additionally, the current benchmarks use synthetic, discrete grid worlds. Future research must address the "entangled" representations found in real-world imagery, where attributes like color and shape are not neatly separated into bits, presenting a significantly harder grounding challenge.

Summary updated

Abstract

Humans can reason compositionally whilst grounding language utterances to the real world. Recent benchmarks like ReaSCAN use navigation tasks grounded in a grid world to assess whether neural models exhibit similar capabilities. In this work, we present a simple transformer-based model that outperforms specialized architectures on ReaSCAN and a modified version of gSCAN. On analyzing the task, we find that identifying the target location in the grid world is the main challenge for the models. Furthermore, we show that a particular split in ReaSCAN, which tests depth generalization, is unfair. On an amended version of this split, we show that transformers can generalize to deeper input structures. Finally, we design a simpler grounded compositional generalization task, RefEx, to investigate how transformers reason compositionally. We show that a single self-attention layer with a single head generalizes to novel combinations of object attributes. Moreover, we derive a precise mathematical construction of the transformer's computations from the learned network. Overall, we provide valuable insights about the grounded compositional generalization task and the behaviour of transformers on it, which would be useful for researchers working in this area.
BibTeX
@inproceedings{sikarwar2022when,
  author = {Ankur Sikarwar and Arkil Patel and Navin Goyal},
  title = {{When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=2pRp19SM8t1},
}
Figure