The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber

Abstract

Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyper-parameters from standard tasks, are shown to fail dramatically. Here we demonstrate that by revisiting model configurations as basic as scaling of embeddings, early stopping, relative positional embedding, and Universal Transformer variants, we can drastically improve the performance of Transformers on systematic generalization. We report improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics dataset. Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS. On SCAN, relative positional embedding largely mitigates the EOS decision problem (Newman et al., 2020), yielding 100% accuracy on the length split with a cutoff at 26. Importantly, performance differences between these models are typically invisible on the IID data split. This calls for proper generalization validation sets for developing neural networks that generalize systematically. We publicly release the code to reproduce our results.

Introduction

Systematic generalization—the ability to extrapolate compositional rules to novel, out-of-distribution (OOD) scenarios—is a hallmark of human intelligence. While Transformers have achieved state-of-the-art results on standard in-distribution (IID) tasks, they have been widely criticized for failing on systematic benchmarks like SCAN or COGS. Previous research often attributed these failures to fundamental architectural limitations, prompting the development of specialized, complex models. This paper challenges that narrative, demonstrating that standard Transformers, when properly configured, can achieve remarkable generalization performance.

Methods: Four Simple Tricks

The authors identify four key configuration details that significantly impact a model's ability to generalize:

  1. Relative Positional Embeddings (RPE): Unlike absolute positional embeddings, which overfit to specific sequence lengths seen during training, RPE focuses on the relative distance between tokens. This makes the model more robust to longer sequences at test time.
  2. Universal Transformers: By sharing weights across layers, Universal Transformers learn iterative operations rather than layer-specific features, which is better suited for algorithmic and compositional tasks.
  3. Position Embedding Downscaling (PED): Standard initialization often scales up word embeddings. The authors found that scaling down positional embeddings using Kaiming initialization (PED) allows the model to prioritize semantic content while still utilizing positional information, leading to better generalization.
  4. Revised Model Selection: The authors argue against standard early stopping based on IID validation sets. Generalization performance often continues to improve long after IID accuracy has reached 100%.

Key Insights and Ideas

The IID Trap and Early Stopping

A critical finding is that IID performance is a poor proxy for systematic generalization. Models that appear identical on an IID validation set can exhibit vastly different OOD performance.

Generalization accuracy continues to improve long after IID performance has saturated, highlighting the danger of early stopping.

As shown in the figure above, generalization accuracy for various scaling methods continues to climb significantly after the initial training phase where IID performance typically plateaus (indicated by the vertical dashed lines). Stopping training early based on IID metrics would result in picking a model with significantly lower generalization capability.

Loss is a Liar

In many systematic generalization tasks, the relationship between loss and accuracy is non-intuitive. On certain datasets, the authors observed that test accuracy can continue to improve even as the validation loss begins to rise.

Test accuracy can improve even as validation loss increases, suggesting that loss is an unreliable metric for model selection in systematic generalization tasks.

This phenomenon, illustrated in the scatter plot above, shows points moving toward higher loss and higher accuracy as training progresses (indicated by the color shift from dark to light). This suggests that using loss for model selection or early stopping can be misleading, potentially leading researchers to discard models that are actually learning the underlying logic of the task more effectively.

The EOS Decision Problem

The failure of models on longer sequences is often linked to the "End-Of-Sentence" (EOS) token. Absolute positional embeddings cause the model to learn a fixed "length budget." Switching to RPE effectively solves this, allowing the model to generalize to lengths far beyond the training distribution.

Key Results

The application of these simple tricks leads to dramatic performance improvements across standard benchmarks:

  • SCAN (Length Split): Accuracy improved from 0% to 100% using RPE.
  • COGS: Performance rose from 35% to 81%.
  • PCFG (Productivity Split): Accuracy increased from 50% to 85%.
  • Mathematics (Add/Sub): Achieved 97% accuracy, surpassing many specialized architectures.

Conclusion

The "failure" of Transformers in systematic generalization is often a failure of configuration and evaluation rather than architecture. By paying attention to "details" like embedding scaling, positional encoding, and training duration, standard Transformers can reach or exceed the performance of specialized models. These findings emphasize the need for the community to adopt proper generalization validation sets and move beyond IID-centric evaluation.

Summary updated

Abstract

Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyper-parameters from standard tasks, are shown to fail dramatically. Here we demonstrate that by revisiting model configurations as basic as scaling of embeddings, early stopping, relative positional embedding, and Universal Transformer variants, we can drastically improve the performance of Transformers on systematic generalization. We report improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics dataset. Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS. On SCAN, relative positional embedding largely mitigates the EOS decision problem (Newman et al., 2020), yielding 100% accuracy on the length split with a cutoff at 26. Importantly, performance differences between these models are typically invisible on the IID data split. This calls for proper generalization validation sets for developing neural networks that generalize systematically. We publicly release the code to reproduce our results.
BibTeX
@inproceedings{csordas2021devil,
  author = {Róbert Csordás and Kazuki Irie and Jürgen Schmidhuber},
  title = {{The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=o-6p-QDZ7nD},
}
Figure