Abstract
Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyper-parameters from standard tasks, are shown to fail dramatically. Here we demonstrate that by revisiting model configurations as basic as scaling of embeddings, early stopping, relative positional embedding, and Universal Transformer variants, we can drastically improve the performance of Transformers on systematic generalization. We report improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics dataset. Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS. On SCAN, relative positional embedding largely mitigates the EOS decision problem (Newman et al., 2020), yielding 100% accuracy on the length split with a cutoff at 26. Importantly, performance differences between these models are typically invisible on the IID data split. This calls for proper generalization validation sets for developing neural networks that generalize systematically. We publicly release the code to reproduce our results.
Introduction
Systematic generalization—the ability to extrapolate compositional rules to novel, out-of-distribution (OOD) scenarios—is a hallmark of human intelligence. While Transformers have achieved state-of-the-art results on standard in-distribution (IID) tasks, they have been widely criticized for failing on systematic benchmarks like SCAN or COGS. Previous research often attributed these failures to fundamental architectural limitations, prompting the development of specialized, complex models. This paper challenges that narrative, demonstrating that standard Transformers, when properly configured, can achieve remarkable generalization performance.
Methods: Four Simple Tricks
The authors identify four key configuration details that significantly impact a model's ability to generalize:
- Relative Positional Embeddings (RPE): Unlike absolute positional embeddings, which overfit to specific sequence lengths seen during training, RPE focuses on the relative distance between tokens. This makes the model more robust to longer sequences at test time.
- Universal Transformers: By sharing weights across layers, Universal Transformers learn iterative operations rather than layer-specific features, which is better suited for algorithmic and compositional tasks.
- Position Embedding Downscaling (PED): Standard initialization often scales up word embeddings. The authors found that scaling down positional embeddings using Kaiming initialization (PED) allows the model to prioritize semantic content while still utilizing positional information, leading to better generalization.
- Revised Model Selection: The authors argue against standard early stopping based on IID validation sets. Generalization performance often continues to improve long after IID accuracy has reached 100%.
Key Insights and Ideas
The IID Trap and Early Stopping
A critical finding is that IID performance is a poor proxy for systematic generalization. Models that appear identical on an IID validation set can exhibit vastly different OOD performance.
As shown in the figure above, generalization accuracy for various scaling methods continues to climb significantly after the initial training phase where IID performance typically plateaus (indicated by the vertical dashed lines). Stopping training early based on IID metrics would result in picking a model with significantly lower generalization capability.
Loss is a Liar
In many systematic generalization tasks, the relationship between loss and accuracy is non-intuitive. On certain datasets, the authors observed that test accuracy can continue to improve even as the validation loss begins to rise.
This phenomenon, illustrated in the scatter plot above, shows points moving toward higher loss and higher accuracy as training progresses (indicated by the color shift from dark to light). This suggests that using loss for model selection or early stopping can be misleading, potentially leading researchers to discard models that are actually learning the underlying logic of the task more effectively.
The EOS Decision Problem
The failure of models on longer sequences is often linked to the "End-Of-Sentence" (EOS) token. Absolute positional embeddings cause the model to learn a fixed "length budget." Switching to RPE effectively solves this, allowing the model to generalize to lengths far beyond the training distribution.
Key Results
The application of these simple tricks leads to dramatic performance improvements across standard benchmarks:
- SCAN (Length Split): Accuracy improved from 0% to 100% using RPE.
- COGS: Performance rose from 35% to 81%.
- PCFG (Productivity Split): Accuracy increased from 50% to 85%.
- Mathematics (Add/Sub): Achieved 97% accuracy, surpassing many specialized architectures.
Conclusion
The "failure" of Transformers in systematic generalization is often a failure of configuration and evaluation rather than architecture. By paying attention to "details" like embedding scaling, positional encoding, and training duration, standard Transformers can reach or exceed the performance of specialized models. These findings emphasize the need for the community to adopt proper generalization validation sets and move beyond IID-centric evaluation.
Summary updated