Róbert Csordás, Kazuki Irie, Juergen Schmidhuber
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021)
Abstract
Transformers are frequently criticized for their inability to achieve "systematic generalization"—the capacity to apply learned rules to novel, complex combinations. This work demonstrates that this perceived limitation often stems from sub-optimal baseline configurations rather than inherent architectural flaws. By implementing "simple tricks" such as relative positional embeddings and optimized embedding scaling, we achieve significant performance gains on reasoning benchmarks without the need for complex neuro-symbolic components.
1. Introduction
In human language and logic, we rely on compositionality: the ability to understand novel phrases like "run slowly" by combining the known meanings of "run" and "slowly." While neural networks, particularly Transformers, excel at IID (independent and identically distributed) data, they often fail to extrapolate to out-of-distribution scenarios, such as processing sequences longer than those seen during training or handling new combinations of known rules.
Previous research suggested that Transformers were fundamentally limited in this regard, prompting a shift toward hybrid neuro-symbolic architectures. We argue that standard training practices and hyperparameter choices have inadvertently hindered Transformer generalization, and that well-tuned standard models are far more capable than previously believed.
2. Methods for Enhancing Generalization
We identify several critical configuration choices that significantly impact a model's reasoning capabilities:
2.1 Relative Positional Encoding (RPE)
Standard Transformers use absolute positional encodings, which fail to generalize to sequence lengths exceeding the training maximum because the model has never encountered those specific position indices. RPE instead encodes the relative distance between tokens, a representation that naturally scales to unseen lengths.
2.2 Position Embedding Downscaling (PED)
The interaction between word and positional embeddings is crucial. We propose Position Embedding Downscaling (PED), which utilizes Kaiming initialization for word embeddings while scaling down positional components. This prevents positional signals from "overwhelming" the semantic content of the embeddings, which is vital for maintaining the priority of token meanings during reasoning.
2.3 Universal Transformers (UT)
Universal Transformers employ weight-sharing across layers, acting as an inductive bias for algorithmic tasks by allowing the model to iteratively refine its internal representations, similar to a recursive function.
3. The Validation Trap and Model Selection
A critical finding is the disconnect between IID validation performance and systematic generalization. Models often reach 100% IID accuracy long before they learn underlying systematic rules. Furthermore, we observe "epoch-wise double descent" where test loss may increase while generalization accuracy continues to improve. Consequently, standard early stopping based on loss or IID accuracy can lead to sub-optimal model selection for reasoning tasks. We argue that specific generalization validation sets are mandatory for measuring real progress.
4. Experimental Results
We evaluated these techniques across five benchmarks: SCAN, CFQ, PCFG, COGS, and the Mathematics Dataset. The improvements were dramatic across the board:
- SCAN (Length split): Relative positional embeddings enabled 100% accuracy on sequences longer than the training set (up to a cutoff of 26).
- COGS: Accuracy jumped from a baseline of 35% to 81%.
- PCFG (Productivity split): Accuracy increased from 50% to 85%.
- CFQ: Universal Transformer variants outperformed previous state-of-the-art results on several splits.
5. Conclusion
Our results suggest that the "devil is in the detail": simple architectural and training adjustments can bridge the gap between neural and symbolic reasoning. By moving beyond sub-optimal baselines and adopting more rigorous generalization validation, the community can unlock the true potential of Transformer architectures for complex reasoning tasks.
Summary updated