Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN

Rahma Chaabouni, Roberto Dessì, Eugene Kharitonov

Introduction

Compositional generalization—the ability to understand and produce novel combinations of known rules—is a hallmark of human intelligence. While modern Transformer-based models dominate natural language processing (NLP), they often struggle with this fundamental logic. This study investigates whether architectural modifications designed to improve performance on diagnostic logic benchmarks, such as SCAN, can translate into tangible gains for real-world machine translation (MT) tasks.

The Compositional Gap in Transformers

Standard sequence-to-sequence Transformers are remarkably effective at translation but surprisingly brittle when faced with systematic rule application. The SCAN benchmark highlights this weakness by requiring models to translate simple command instructions (e.g., "run twice") into action sequences. While humans intuitively generalize from "run twice" to "jump twice," vanilla Transformers frequently fail these tests. This raises a critical question: if Transformers lack this basic systematicity, why do they succeed in translation, and can "fixing" their logic improve their linguistic performance?

Architectural Enhancements: Locality and Gating

Drawing inspiration from Convolutional Seq2Seq (ConvS2S) models, which excel at SCAN, researchers identified two missing components in standard Transformers: localized receptive fields and gating mechanisms. They proposed several "focused modifications":

  • Self-Attention Gate (SAG): A learned scalar parameter that allows the model to modulate the signal from the self-attention layer.
  • Fixed-Span Self-Attention: A hard constraint forcing the model to attend only to tokens within a local window.
  • T5 Attention: The use of learned relative position biases to encourage a preference for local context while maintaining global reach.

Experimental Setup

The models were evaluated across four domains:

  1. SCAN Benchmark: Testing splits like jump and around-right.
  2. Standard MT: German-to-English (IWSLT’14).
  3. Low-Resource MT: English to/from Nepali and Sinhala (FLoRes).
  4. Noun-Adjective Ordering: A novel English-French task probing the generalization of adjective placement.

Key Results

The architectural modifications yielded significant improvements in systematic reasoning. The hybrid SAG + T5 model increased accuracy on the challenging "SCAN-jump" split from a dismal 3.4% to 43.0%.

Performance on SCAN splits across different model architectures.

These logic-based improvements translated directly to natural language tasks, particularly in data-scarce environments. In low-resource MT (FLoRes), modified models saw BLEU score increases of up to 13.1%. In the French noun-adjective task, the models showed a 14% improvement in correctly placing adjectives, demonstrating better structural generalization.

Qualitative comparison of translation outputs between vanilla Transformer and SAG+T5.

Discussion and Conclusion

The findings suggest that systematicity is not just an abstract requirement but a practical necessity for low-resource NLP. By introducing inductive biases for locality and gating, models become less reliant on memorizing patterns and more capable of rule-based reasoning. While a gap remains between these models and human-level generalization, these modifications provide a promising path toward more robust and logical language models.

Summary updated

BibTeX
@inproceedings{chaabouni2021can,
  author = {Rahma Chaabouni and Roberto Dessì and Eugene Kharitonov},
  title = {{Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN}},
  booktitle = {BlackboxNLP @ EMNLP},
  year = {2021},
  url = {https://openreview.net/forum?id=NsTsDov9g-},
}
Figure