Introduction
Compositional generalization is a hallmark of human intelligence—the ability to understand novel combinations of known components. For instance, if a person understands the concept of a "red cube" and the action of "spinning," they can immediately interpret the command to "spin the red cube," even if they have never witnessed that specific combination before.
In the domain of grounded language understanding, artificial intelligence models must map natural language commands to specific actions within a visual environment. While humans perform these tasks intuitively, neural models frequently struggle with "out-of-distribution" (OOD) scenarios where the logic or attribute combinations differ from the training data. This paper investigates whether generic Transformer architectures can overcome these hurdles and identifies the specific bottlenecks preventing generalization.
The GroCoT Architecture
The authors introduce GroCoT (Grounded Compositional Transformer), a model that applies three targeted enhancements to the standard multi-modal Transformer to improve its reasoning capabilities:
- Improved Spatial Representation: Standard CNNs often have limited receptive fields. GroCoT tokenizes each grid cell individually and incorporates 2D positional encodings, allowing the model to reason about spatial relationships across any distance.
- Interleaving Self-Attention: Instead of immediately mixing vision and language through cross-modal attention, GroCoT interleaves self-attention layers. This allows each modality to develop robust internal representations before synchronization.
- Modified World State Encoding: The model addresses a common flaw in representing overlapping objects (e.g., a cylinder inside a box). By ensuring the encoding can disambiguate the attributes of a container from its contents, the model avoids confusion during complex spatial reasoning.
Benchmarking Compositional Logic
The study evaluates GroCoT across several established and novel benchmarks designed to test different facets of compositional logic. These include gSCAN and GSRR for navigation, and ReaSCAN, which introduces relative clauses and complex spatial relations.
To further probe the model's limits, the authors introduced C2-deeper, a split designed to test generalization to deeper command structures (e.g., multiple nested "that is" clauses). They also created RefEx, a simplified "Referring Expressions" task used to isolate target identification from action execution for mathematical analysis.
Key Insights and Results
Target Identification is the Primary Bottleneck
Through "gold target" experiments—where the model is explicitly told the target's location—the authors found that navigation accuracy jumps to nearly 100%. This proves that the failure of current models lies almost entirely in identifying which object the command refers to, rather than the subsequent planning of actions.
State-of-the-Art Performance
GroCoT significantly outperformed previous specialized architectures. On the GSRR benchmark, it achieved 98.8% accuracy, a massive leap from the previous best of 63.5%. On the complex ReaSCAN benchmark, it reached 82.2% accuracy. Furthermore, on the C2-deeper split, GroCoT achieved 85.6%, demonstrating that Transformers can handle recursive logic if the training distribution is fair.
Mathematical Transparency
Using the RefEx task, the authors provided a rare look into the "black box" of Transformer logic. They proved that a single-layer, single-head attention mechanism is sufficient to ground attributes. By deriving a mathematical construction from learned weights, they showed the model performs a series of dot products that effectively act as "attribute filters" to isolate the target object.
Conclusion and Future Work
While GroCoT excels at target identification and recursive logic, challenges remain. The model still struggles with certain gSCAN splits that require generalizing to entirely novel output sequences (e.g., moving in a direction never seen during training). Additionally, the current benchmarks use synthetic, discrete grid worlds. Future research must address the "entangled" representations found in real-world imagery, where attributes like color and shape are not neatly separated into bits, presenting a significantly harder grounding challenge.
Summary updated