The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber

Róbert Csordás, Kazuki Irie, Juergen Schmidhuber
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021)

Abstract

Transformers are frequently criticized for their inability to achieve "systematic generalization"—the capacity to apply learned rules to novel, complex combinations. This work demonstrates that this perceived limitation often stems from sub-optimal baseline configurations rather than inherent architectural flaws. By implementing "simple tricks" such as relative positional embeddings and optimized embedding scaling, we achieve significant performance gains on reasoning benchmarks without the need for complex neuro-symbolic components.

1. Introduction

In human language and logic, we rely on compositionality: the ability to understand novel phrases like "run slowly" by combining the known meanings of "run" and "slowly." While neural networks, particularly Transformers, excel at IID (independent and identically distributed) data, they often fail to extrapolate to out-of-distribution scenarios, such as processing sequences longer than those seen during training or handling new combinations of known rules.

Previous research suggested that Transformers were fundamentally limited in this regard, prompting a shift toward hybrid neuro-symbolic architectures. We argue that standard training practices and hyperparameter choices have inadvertently hindered Transformer generalization, and that well-tuned standard models are far more capable than previously believed.

2. Methods for Enhancing Generalization

We identify several critical configuration choices that significantly impact a model's reasoning capabilities:

2.1 Relative Positional Encoding (RPE)

Standard Transformers use absolute positional encodings, which fail to generalize to sequence lengths exceeding the training maximum because the model has never encountered those specific position indices. RPE instead encodes the relative distance between tokens, a representation that naturally scales to unseen lengths.

2.2 Position Embedding Downscaling (PED)

The interaction between word and positional embeddings is crucial. We propose Position Embedding Downscaling (PED), which utilizes Kaiming initialization for word embeddings while scaling down positional components. This prevents positional signals from "overwhelming" the semantic content of the embeddings, which is vital for maintaining the priority of token meanings during reasoning.

Generalization accuracy over training steps for different scaling methods. PED (green) significantly outperforms standard scaling (blue) and token embedding upscaling (orange).

2.3 Universal Transformers (UT)

Universal Transformers employ weight-sharing across layers, acting as an inductive bias for algorithmic tasks by allowing the model to iteratively refine its internal representations, similar to a recursive function.

3. The Validation Trap and Model Selection

A critical finding is the disconnect between IID validation performance and systematic generalization. Models often reach 100% IID accuracy long before they learn underlying systematic rules. Furthermore, we observe "epoch-wise double descent" where test loss may increase while generalization accuracy continues to improve. Consequently, standard early stopping based on loss or IID accuracy can lead to sub-optimal model selection for reasoning tasks. We argue that specific generalization validation sets are mandatory for measuring real progress.

4. Experimental Results

We evaluated these techniques across five benchmarks: SCAN, CFQ, PCFG, COGS, and the Mathematics Dataset. The improvements were dramatic across the board:

  • SCAN (Length split): Relative positional embeddings enabled 100% accuracy on sequences longer than the training set (up to a cutoff of 26).
  • COGS: Accuracy jumped from a baseline of 35% to 81%.
  • PCFG (Productivity split): Accuracy increased from 50% to 85%.
  • CFQ: Universal Transformer variants outperformed previous state-of-the-art results on several splits.
Performance on SCAN length splits. The inclusion of Relative PE allows both standard and Universal Transformers to achieve perfect generalization (1.00) across various length cutoffs.

5. Conclusion

Our results suggest that the "devil is in the detail": simple architectural and training adjustments can bridge the gap between neural and symbolic reasoning. By moving beyond sub-optimal baselines and adopting more rigorous generalization validation, the community can unlock the true potential of Transformer architectures for complex reasoning tasks.

Summary updated

BibTeX
@inproceedings{csordas2021devil,
  author = {Róbert Csordás and Kazuki Irie and Jürgen Schmidhuber},
  title = {{The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=o-6p-QDZ7nD},
}
Figure